gistfile1.txt

ureFreeSpace(2746) called with curMem=18694915, maxMem=4042889625
16/09/07 19:10:22 INFO MemoryStore: Block broadcast_970_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:22 INFO BlockManagerInfo: Added broadcast_970_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:22 INFO SparkContext: Created broadcast 970 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:22 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 641 (MapPartitionsRDD[1987] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:22 INFO TaskSchedulerImpl: Adding task set 641.0 with 8 tasks
16/09/07 19:10:22 INFO TaskSetManager: Starting task 0.0 in stage 641.0 (TID 5128, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:22 INFO TaskSetManager: Starting task 1.0 in stage 641.0 (TID 5129, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:22 INFO TaskSetManager: Starting task 2.0 in stage 641.0 (TID 5130, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:22 INFO TaskSetManager: Starting task 3.0 in stage 641.0 (TID 5131, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:22 INFO TaskSetManager: Starting task 4.0 in stage 641.0 (TID 5132, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:22 INFO TaskSetManager: Starting task 5.0 in stage 641.0 (TID 5133, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:22 INFO TaskSetManager: Starting task 6.0 in stage 641.0 (TID 5134, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:22 INFO TaskSetManager: Starting task 7.0 in stage 641.0 (TID 5135, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:22 INFO Executor: Running task 2.0 in stage 641.0 (TID 5130)
16/09/07 19:10:22 INFO Executor: Running task 5.0 in stage 641.0 (TID 5133)
16/09/07 19:10:22 INFO Executor: Running task 3.0 in stage 641.0 (TID 5131)
16/09/07 19:10:22 INFO Executor: Running task 6.0 in stage 641.0 (TID 5134)
16/09/07 19:10:22 INFO Executor: Running task 4.0 in stage 641.0 (TID 5132)
16/09/07 19:10:22 INFO Executor: Running task 1.0 in stage 641.0 (TID 5129)
16/09/07 19:10:22 INFO Executor: Running task 7.0 in stage 641.0 (TID 5135)
16/09/07 19:10:22 INFO Executor: Running task 0.0 in stage 641.0 (TID 5128)
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:22 INFO Executor: Finished task 5.0 in stage 641.0 (TID 5133). 2160 bytes result sent to driver
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:22 INFO TaskSetManager: Finished task 5.0 in stage 641.0 (TID 5133) in 1 ms on localhost (1/8)
16/09/07 19:10:22 INFO Executor: Finished task 1.0 in stage 641.0 (TID 5129). 2160 bytes result sent to driver
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:22 INFO Executor: Finished task 4.0 in stage 641.0 (TID 5132). 2160 bytes result sent to driver
16/09/07 19:10:22 INFO TaskSetManager: Finished task 1.0 in stage 641.0 (TID 5129) in 2 ms on localhost (2/8)
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:22 INFO Executor: Finished task 2.0 in stage 641.0 (TID 5130). 2160 bytes result sent to driver
16/09/07 19:10:22 INFO TaskSetManager: Finished task 4.0 in stage 641.0 (TID 5132) in 2 ms on localhost (3/8)
16/09/07 19:10:22 INFO TaskSetManager: Finished task 2.0 in stage 641.0 (TID 5130) in 2 ms on localhost (4/8)
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:22 INFO Executor: Finished task 3.0 in stage 641.0 (TID 5131). 2160 bytes result sent to driver
16/09/07 19:10:22 INFO TaskSetManager: Finished task 3.0 in stage 641.0 (TID 5131) in 3 ms on localhost (5/8)
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:22 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:22 INFO BlockManagerInfo: Removed broadcast_969_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:23 INFO Executor: Finished task 6.0 in stage 641.0 (TID 5134). 3485274 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 6.0 in stage 641.0 (TID 5134) in 774 ms on localhost (6/8)
16/09/07 19:10:23 INFO Executor: Finished task 7.0 in stage 641.0 (TID 5135). 3485274 bytes result sent to driver
16/09/07 19:10:23 INFO Executor: Finished task 0.0 in stage 641.0 (TID 5128). 3485274 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 7.0 in stage 641.0 (TID 5135) in 819 ms on localhost (7/8)
16/09/07 19:10:23 INFO TaskSetManager: Finished task 0.0 in stage 641.0 (TID 5128) in 830 ms on localhost (8/8)
16/09/07 19:10:23 INFO TaskSchedulerImpl: Removed TaskSet 641.0, whose tasks have all completed, from pool 
16/09/07 19:10:23 INFO DAGScheduler: ResultStage 641 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.830 s
16/09/07 19:10:23 INFO DAGScheduler: Job 325 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.919092 s
16/09/07 19:10:23 INFO ParameterAveragingTrainingMaster: Completed training of split 4 of 78
16/09/07 19:10:23 INFO ParameterAveragingTrainingMaster: Starting training of split 5 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:23 INFO BlockManagerInfo: Removed broadcast_970_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:23 INFO ContextCleaner: Cleaned accumulator 642
16/09/07 19:10:23 INFO ContextCleaner: Cleaned accumulator 641
16/09/07 19:10:23 INFO BlockManagerInfo: Removed broadcast_968_piece0 on localhost:41251 in memory (size: 270.0 KB, free: 3.8 GB)
16/09/07 19:10:23 INFO ContextCleaner: Cleaned shuffle 315
16/09/07 19:10:23 INFO MemoryStore: ensureFreeSpace(17577256) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:23 INFO MemoryStore: Block broadcast_971 stored as values in memory (estimated size 16.8 MB, free 3.7 GB)
16/09/07 19:10:23 INFO MemoryStore: ensureFreeSpace(276187) called with curMem=18121080, maxMem=4042889625
16/09/07 19:10:23 INFO MemoryStore: Block broadcast_971_piece0 stored as bytes in memory (estimated size 269.7 KB, free 3.7 GB)
16/09/07 19:10:23 INFO BlockManagerInfo: Added broadcast_971_piece0 in memory on localhost:41251 (size: 269.7 KB, free: 3.8 GB)
16/09/07 19:10:23 INFO SparkContext: Created broadcast 971 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:23 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:23 INFO DAGScheduler: Registering RDD 1988 (repartition at SparkUtils.java:190)
16/09/07 19:10:23 INFO DAGScheduler: Got job 326 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:23 INFO DAGScheduler: Final stage: ResultStage 643(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 642)
16/09/07 19:10:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 642)
16/09/07 19:10:23 INFO DAGScheduler: Submitting ShuffleMapStage 642 (MapPartitionsRDD[1988] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:23 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18397267, maxMem=4042889625
16/09/07 19:10:23 INFO MemoryStore: Block broadcast_972 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:23 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18401731, maxMem=4042889625
16/09/07 19:10:23 INFO MemoryStore: Block broadcast_972_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:23 INFO BlockManagerInfo: Added broadcast_972_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:23 INFO SparkContext: Created broadcast 972 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:23 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 642 (MapPartitionsRDD[1988] at repartition at SparkUtils.java:190)
16/09/07 19:10:23 INFO TaskSchedulerImpl: Adding task set 642.0 with 8 tasks
16/09/07 19:10:23 INFO TaskSetManager: Starting task 0.0 in stage 642.0 (TID 5136, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 1.0 in stage 642.0 (TID 5137, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 2.0 in stage 642.0 (TID 5138, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 3.0 in stage 642.0 (TID 5139, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 4.0 in stage 642.0 (TID 5140, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 5.0 in stage 642.0 (TID 5141, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 6.0 in stage 642.0 (TID 5142, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 7.0 in stage 642.0 (TID 5143, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:23 INFO Executor: Running task 2.0 in stage 642.0 (TID 5138)
16/09/07 19:10:23 INFO Executor: Running task 0.0 in stage 642.0 (TID 5136)
16/09/07 19:10:23 INFO Executor: Running task 3.0 in stage 642.0 (TID 5139)
16/09/07 19:10:23 INFO Executor: Running task 5.0 in stage 642.0 (TID 5141)
16/09/07 19:10:23 INFO Executor: Running task 6.0 in stage 642.0 (TID 5142)
16/09/07 19:10:23 INFO Executor: Running task 1.0 in stage 642.0 (TID 5137)
16/09/07 19:10:23 INFO Executor: Running task 7.0 in stage 642.0 (TID 5143)
16/09/07 19:10:23 INFO Executor: Running task 4.0 in stage 642.0 (TID 5140)
16/09/07 19:10:23 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:23 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:23 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:23 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:23 INFO Executor: Finished task 5.0 in stage 642.0 (TID 5141). 2260 bytes result sent to driver
16/09/07 19:10:23 INFO Executor: Finished task 6.0 in stage 642.0 (TID 5142). 2260 bytes result sent to driver
16/09/07 19:10:23 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:23 INFO TaskSetManager: Finished task 5.0 in stage 642.0 (TID 5141) in 2 ms on localhost (1/8)
16/09/07 19:10:23 INFO TaskSetManager: Finished task 6.0 in stage 642.0 (TID 5142) in 2 ms on localhost (2/8)
16/09/07 19:10:23 INFO Executor: Finished task 1.0 in stage 642.0 (TID 5137). 2260 bytes result sent to driver
16/09/07 19:10:23 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:23 INFO Executor: Finished task 3.0 in stage 642.0 (TID 5139). 2260 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 1.0 in stage 642.0 (TID 5137) in 3 ms on localhost (3/8)
16/09/07 19:10:23 INFO Executor: Finished task 7.0 in stage 642.0 (TID 5143). 2260 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 3.0 in stage 642.0 (TID 5139) in 3 ms on localhost (4/8)
16/09/07 19:10:23 INFO TaskSetManager: Finished task 7.0 in stage 642.0 (TID 5143) in 3 ms on localhost (5/8)
16/09/07 19:10:23 INFO Executor: Finished task 4.0 in stage 642.0 (TID 5140). 2260 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 4.0 in stage 642.0 (TID 5140) in 3 ms on localhost (6/8)
16/09/07 19:10:23 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:23 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:23 INFO Executor: Finished task 2.0 in stage 642.0 (TID 5138). 2260 bytes result sent to driver
16/09/07 19:10:23 INFO Executor: Finished task 0.0 in stage 642.0 (TID 5136). 2260 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 2.0 in stage 642.0 (TID 5138) in 5 ms on localhost (7/8)
16/09/07 19:10:23 INFO TaskSetManager: Finished task 0.0 in stage 642.0 (TID 5136) in 5 ms on localhost (8/8)
16/09/07 19:10:23 INFO TaskSchedulerImpl: Removed TaskSet 642.0, whose tasks have all completed, from pool 
16/09/07 19:10:23 INFO DAGScheduler: ShuffleMapStage 642 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:23 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:23 INFO DAGScheduler: running: Set()
16/09/07 19:10:23 INFO DAGScheduler: waiting: Set(ResultStage 643)
16/09/07 19:10:23 INFO DAGScheduler: failed: Set()
16/09/07 19:10:23 INFO DAGScheduler: Missing parents for ResultStage 643: List()
16/09/07 19:10:23 INFO DAGScheduler: Submitting ResultStage 643 (MapPartitionsRDD[1992] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:23 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18404295, maxMem=4042889625
16/09/07 19:10:23 INFO MemoryStore: Block broadcast_973 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:23 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18409391, maxMem=4042889625
16/09/07 19:10:23 INFO MemoryStore: Block broadcast_973_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:23 INFO BlockManagerInfo: Added broadcast_973_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:23 INFO SparkContext: Created broadcast 973 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:23 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 643 (MapPartitionsRDD[1992] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:23 INFO TaskSchedulerImpl: Adding task set 643.0 with 8 tasks
16/09/07 19:10:23 INFO TaskSetManager: Starting task 0.0 in stage 643.0 (TID 5144, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 1.0 in stage 643.0 (TID 5145, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 2.0 in stage 643.0 (TID 5146, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 3.0 in stage 643.0 (TID 5147, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 4.0 in stage 643.0 (TID 5148, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 5.0 in stage 643.0 (TID 5149, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 6.0 in stage 643.0 (TID 5150, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:23 INFO TaskSetManager: Starting task 7.0 in stage 643.0 (TID 5151, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:23 INFO Executor: Running task 1.0 in stage 643.0 (TID 5145)
16/09/07 19:10:23 INFO Executor: Running task 5.0 in stage 643.0 (TID 5149)
16/09/07 19:10:23 INFO Executor: Running task 7.0 in stage 643.0 (TID 5151)
16/09/07 19:10:23 INFO Executor: Running task 3.0 in stage 643.0 (TID 5147)
16/09/07 19:10:23 INFO Executor: Running task 6.0 in stage 643.0 (TID 5150)
16/09/07 19:10:23 INFO Executor: Running task 0.0 in stage 643.0 (TID 5144)
16/09/07 19:10:23 INFO Executor: Running task 4.0 in stage 643.0 (TID 5148)
16/09/07 19:10:23 INFO Executor: Running task 2.0 in stage 643.0 (TID 5146)
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:23 INFO Executor: Finished task 1.0 in stage 643.0 (TID 5145). 2160 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 1.0 in stage 643.0 (TID 5145) in 1 ms on localhost (1/8)
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:23 INFO Executor: Finished task 2.0 in stage 643.0 (TID 5146). 2160 bytes result sent to driver
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:23 INFO Executor: Finished task 4.0 in stage 643.0 (TID 5148). 2160 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 2.0 in stage 643.0 (TID 5146) in 2 ms on localhost (2/8)
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:23 INFO TaskSetManager: Finished task 4.0 in stage 643.0 (TID 5148) in 2 ms on localhost (3/8)
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:23 INFO Executor: Finished task 5.0 in stage 643.0 (TID 5149). 2160 bytes result sent to driver
16/09/07 19:10:23 INFO Executor: Finished task 3.0 in stage 643.0 (TID 5147). 2160 bytes result sent to driver
16/09/07 19:10:23 INFO TaskSetManager: Finished task 5.0 in stage 643.0 (TID 5149) in 3 ms on localhost (4/8)
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:23 INFO TaskSetManager: Finished task 3.0 in stage 643.0 (TID 5147) in 3 ms on localhost (5/8)
16/09/07 19:10:23 INFO BlockManagerInfo: Removed broadcast_972_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:24 INFO Executor: Finished task 0.0 in stage 643.0 (TID 5144). 3485274 bytes result sent to driver
16/09/07 19:10:24 INFO Executor: Finished task 6.0 in stage 643.0 (TID 5150). 3485274 bytes result sent to driver
16/09/07 19:10:24 INFO Executor: Finished task 7.0 in stage 643.0 (TID 5151). 3485274 bytes result sent to driver
16/09/07 19:10:24 INFO TaskSetManager: Finished task 6.0 in stage 643.0 (TID 5150) in 893 ms on localhost (6/8)
16/09/07 19:10:24 INFO TaskSetManager: Finished task 0.0 in stage 643.0 (TID 5144) in 902 ms on localhost (7/8)
16/09/07 19:10:24 INFO TaskSetManager: Finished task 7.0 in stage 643.0 (TID 5151) in 902 ms on localhost (8/8)
16/09/07 19:10:24 INFO TaskSchedulerImpl: Removed TaskSet 643.0, whose tasks have all completed, from pool 
16/09/07 19:10:24 INFO DAGScheduler: ResultStage 643 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.902 s
16/09/07 19:10:24 INFO DAGScheduler: Job 326 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.912190 s
16/09/07 19:10:24 INFO ParameterAveragingTrainingMaster: Completed training of split 5 of 78
16/09/07 19:10:24 INFO ParameterAveragingTrainingMaster: Starting training of split 6 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:24 INFO MemoryStore: ensureFreeSpace(17883504) called with curMem=18405106, maxMem=4042889625
16/09/07 19:10:24 INFO MemoryStore: Block broadcast_974 stored as values in memory (estimated size 17.1 MB, free 3.7 GB)
16/09/07 19:10:24 INFO BlockManagerInfo: Removed broadcast_973_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:24 INFO ContextCleaner: Cleaned accumulator 644
16/09/07 19:10:24 INFO ContextCleaner: Cleaned accumulator 643
16/09/07 19:10:24 INFO BlockManagerInfo: Removed broadcast_971_piece0 on localhost:41251 in memory (size: 269.7 KB, free: 3.8 GB)
16/09/07 19:10:24 INFO ContextCleaner: Cleaned shuffle 316
16/09/07 19:10:24 INFO MemoryStore: ensureFreeSpace(276941) called with curMem=18427328, maxMem=4042889625
16/09/07 19:10:24 INFO MemoryStore: Block broadcast_974_piece0 stored as bytes in memory (estimated size 270.5 KB, free 3.7 GB)
16/09/07 19:10:24 INFO BlockManagerInfo: Added broadcast_974_piece0 in memory on localhost:41251 (size: 270.5 KB, free: 3.8 GB)
16/09/07 19:10:24 INFO SparkContext: Created broadcast 974 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:24 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:24 INFO DAGScheduler: Registering RDD 1993 (repartition at SparkUtils.java:190)
16/09/07 19:10:24 INFO DAGScheduler: Got job 327 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:24 INFO DAGScheduler: Final stage: ResultStage 645(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 644)
16/09/07 19:10:24 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 644)
16/09/07 19:10:24 INFO DAGScheduler: Submitting ShuffleMapStage 644 (MapPartitionsRDD[1993] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:24 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18704269, maxMem=4042889625
16/09/07 19:10:24 INFO MemoryStore: Block broadcast_975 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:24 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18708733, maxMem=4042889625
16/09/07 19:10:24 INFO MemoryStore: Block broadcast_975_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:24 INFO BlockManagerInfo: Added broadcast_975_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:24 INFO SparkContext: Created broadcast 975 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:24 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 644 (MapPartitionsRDD[1993] at repartition at SparkUtils.java:190)
16/09/07 19:10:24 INFO TaskSchedulerImpl: Adding task set 644.0 with 8 tasks
16/09/07 19:10:24 INFO TaskSetManager: Starting task 0.0 in stage 644.0 (TID 5152, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 1.0 in stage 644.0 (TID 5153, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 2.0 in stage 644.0 (TID 5154, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 3.0 in stage 644.0 (TID 5155, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 4.0 in stage 644.0 (TID 5156, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 5.0 in stage 644.0 (TID 5157, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 6.0 in stage 644.0 (TID 5158, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 7.0 in stage 644.0 (TID 5159, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:24 INFO Executor: Running task 1.0 in stage 644.0 (TID 5153)
16/09/07 19:10:24 INFO Executor: Running task 0.0 in stage 644.0 (TID 5152)
16/09/07 19:10:24 INFO Executor: Running task 6.0 in stage 644.0 (TID 5158)
16/09/07 19:10:24 INFO Executor: Running task 4.0 in stage 644.0 (TID 5156)
16/09/07 19:10:24 INFO Executor: Running task 3.0 in stage 644.0 (TID 5155)
16/09/07 19:10:24 INFO Executor: Running task 2.0 in stage 644.0 (TID 5154)
16/09/07 19:10:24 INFO Executor: Running task 5.0 in stage 644.0 (TID 5157)
16/09/07 19:10:24 INFO Executor: Running task 7.0 in stage 644.0 (TID 5159)
16/09/07 19:10:24 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:24 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:24 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:24 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:24 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:24 INFO Executor: Finished task 6.0 in stage 644.0 (TID 5158). 2260 bytes result sent to driver
16/09/07 19:10:24 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:24 INFO TaskSetManager: Finished task 6.0 in stage 644.0 (TID 5158) in 2 ms on localhost (1/8)
16/09/07 19:10:24 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:24 INFO Executor: Finished task 2.0 in stage 644.0 (TID 5154). 2260 bytes result sent to driver
16/09/07 19:10:24 INFO TaskSetManager: Finished task 2.0 in stage 644.0 (TID 5154) in 2 ms on localhost (2/8)
16/09/07 19:10:24 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:24 INFO Executor: Finished task 1.0 in stage 644.0 (TID 5153). 2260 bytes result sent to driver
16/09/07 19:10:24 INFO TaskSetManager: Finished task 1.0 in stage 644.0 (TID 5153) in 3 ms on localhost (3/8)
16/09/07 19:10:24 INFO Executor: Finished task 4.0 in stage 644.0 (TID 5156). 2260 bytes result sent to driver
16/09/07 19:10:24 INFO TaskSetManager: Finished task 4.0 in stage 644.0 (TID 5156) in 3 ms on localhost (4/8)
16/09/07 19:10:24 INFO Executor: Finished task 5.0 in stage 644.0 (TID 5157). 2260 bytes result sent to driver
16/09/07 19:10:24 INFO Executor: Finished task 3.0 in stage 644.0 (TID 5155). 2260 bytes result sent to driver
16/09/07 19:10:24 INFO Executor: Finished task 7.0 in stage 644.0 (TID 5159). 2260 bytes result sent to driver
16/09/07 19:10:24 INFO TaskSetManager: Finished task 3.0 in stage 644.0 (TID 5155) in 3 ms on localhost (5/8)
16/09/07 19:10:24 INFO TaskSetManager: Finished task 5.0 in stage 644.0 (TID 5157) in 3 ms on localhost (6/8)
16/09/07 19:10:24 INFO TaskSetManager: Finished task 7.0 in stage 644.0 (TID 5159) in 3 ms on localhost (7/8)
16/09/07 19:10:24 INFO Executor: Finished task 0.0 in stage 644.0 (TID 5152). 2260 bytes result sent to driver
16/09/07 19:10:24 INFO TaskSetManager: Finished task 0.0 in stage 644.0 (TID 5152) in 5 ms on localhost (8/8)
16/09/07 19:10:24 INFO TaskSchedulerImpl: Removed TaskSet 644.0, whose tasks have all completed, from pool 
16/09/07 19:10:24 INFO DAGScheduler: ShuffleMapStage 644 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:24 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:24 INFO DAGScheduler: running: Set()
16/09/07 19:10:24 INFO DAGScheduler: waiting: Set(ResultStage 645)
16/09/07 19:10:24 INFO DAGScheduler: failed: Set()
16/09/07 19:10:24 INFO DAGScheduler: Missing parents for ResultStage 645: List()
16/09/07 19:10:24 INFO DAGScheduler: Submitting ResultStage 645 (MapPartitionsRDD[1997] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:24 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18711297, maxMem=4042889625
16/09/07 19:10:24 INFO MemoryStore: Block broadcast_976 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:24 INFO MemoryStore: ensureFreeSpace(2742) called with curMem=18716393, maxMem=4042889625
16/09/07 19:10:24 INFO MemoryStore: Block broadcast_976_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:24 INFO BlockManagerInfo: Added broadcast_976_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:24 INFO SparkContext: Created broadcast 976 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:24 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 645 (MapPartitionsRDD[1997] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:24 INFO TaskSchedulerImpl: Adding task set 645.0 with 8 tasks
16/09/07 19:10:24 INFO TaskSetManager: Starting task 0.0 in stage 645.0 (TID 5160, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 1.0 in stage 645.0 (TID 5161, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 2.0 in stage 645.0 (TID 5162, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 3.0 in stage 645.0 (TID 5163, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 4.0 in stage 645.0 (TID 5164, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 5.0 in stage 645.0 (TID 5165, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 6.0 in stage 645.0 (TID 5166, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:24 INFO TaskSetManager: Starting task 7.0 in stage 645.0 (TID 5167, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:24 INFO Executor: Running task 0.0 in stage 645.0 (TID 5160)
16/09/07 19:10:24 INFO Executor: Running task 2.0 in stage 645.0 (TID 5162)
16/09/07 19:10:24 INFO Executor: Running task 1.0 in stage 645.0 (TID 5161)
16/09/07 19:10:24 INFO Executor: Running task 3.0 in stage 645.0 (TID 5163)
16/09/07 19:10:24 INFO Executor: Running task 4.0 in stage 645.0 (TID 5164)
16/09/07 19:10:24 INFO Executor: Running task 5.0 in stage 645.0 (TID 5165)
16/09/07 19:10:24 INFO Executor: Running task 6.0 in stage 645.0 (TID 5166)
16/09/07 19:10:24 INFO Executor: Running task 7.0 in stage 645.0 (TID 5167)
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:24 INFO Executor: Finished task 2.0 in stage 645.0 (TID 5162). 2160 bytes result sent to driver
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:24 INFO Executor: Finished task 5.0 in stage 645.0 (TID 5165). 2160 bytes result sent to driver
16/09/07 19:10:24 INFO TaskSetManager: Finished task 2.0 in stage 645.0 (TID 5162) in 2 ms on localhost (1/8)
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:24 INFO TaskSetManager: Finished task 5.0 in stage 645.0 (TID 5165) in 2 ms on localhost (2/8)
16/09/07 19:10:24 INFO Executor: Finished task 3.0 in stage 645.0 (TID 5163). 2160 bytes result sent to driver
16/09/07 19:10:24 INFO Executor: Finished task 1.0 in stage 645.0 (TID 5161). 2160 bytes result sent to driver
16/09/07 19:10:24 INFO TaskSetManager: Finished task 3.0 in stage 645.0 (TID 5163) in 3 ms on localhost (3/8)
16/09/07 19:10:24 INFO TaskSetManager: Finished task 1.0 in stage 645.0 (TID 5161) in 3 ms on localhost (4/8)
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:24 INFO Executor: Finished task 4.0 in stage 645.0 (TID 5164). 2160 bytes result sent to driver
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:24 INFO TaskSetManager: Finished task 4.0 in stage 645.0 (TID 5164) in 4 ms on localhost (5/8)
16/09/07 19:10:24 INFO BlockManagerInfo: Removed broadcast_975_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:25 INFO Executor: Finished task 6.0 in stage 645.0 (TID 5166). 3485274 bytes result sent to driver
16/09/07 19:10:25 INFO Executor: Finished task 7.0 in stage 645.0 (TID 5167). 3485274 bytes result sent to driver
16/09/07 19:10:25 INFO Executor: Finished task 0.0 in stage 645.0 (TID 5160). 3485274 bytes result sent to driver
16/09/07 19:10:25 INFO TaskSetManager: Finished task 6.0 in stage 645.0 (TID 5166) in 851 ms on localhost (6/8)
16/09/07 19:10:25 INFO TaskSetManager: Finished task 7.0 in stage 645.0 (TID 5167) in 852 ms on localhost (7/8)
16/09/07 19:10:25 INFO TaskSetManager: Finished task 0.0 in stage 645.0 (TID 5160) in 855 ms on localhost (8/8)
16/09/07 19:10:25 INFO TaskSchedulerImpl: Removed TaskSet 645.0, whose tasks have all completed, from pool 
16/09/07 19:10:25 INFO DAGScheduler: ResultStage 645 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.855 s
16/09/07 19:10:25 INFO DAGScheduler: Job 327 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.862848 s
16/09/07 19:10:25 INFO ParameterAveragingTrainingMaster: Completed training of split 6 of 78
16/09/07 19:10:25 INFO ParameterAveragingTrainingMaster: Starting training of split 7 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:25 INFO MemoryStore: ensureFreeSpace(17965496) called with curMem=18712107, maxMem=4042889625
16/09/07 19:10:25 INFO MemoryStore: Block broadcast_977 stored as values in memory (estimated size 17.1 MB, free 3.7 GB)
16/09/07 19:10:25 INFO BlockManagerInfo: Removed broadcast_976_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:25 INFO ContextCleaner: Cleaned accumulator 646
16/09/07 19:10:25 INFO ContextCleaner: Cleaned accumulator 645
16/09/07 19:10:25 INFO BlockManagerInfo: Removed broadcast_974_piece0 on localhost:41251 in memory (size: 270.5 KB, free: 3.8 GB)
16/09/07 19:10:25 INFO ContextCleaner: Cleaned shuffle 317
16/09/07 19:10:25 INFO MemoryStore: ensureFreeSpace(276601) called with curMem=18509320, maxMem=4042889625
16/09/07 19:10:25 INFO MemoryStore: Block broadcast_977_piece0 stored as bytes in memory (estimated size 270.1 KB, free 3.7 GB)
16/09/07 19:10:25 INFO BlockManagerInfo: Added broadcast_977_piece0 in memory on localhost:41251 (size: 270.1 KB, free: 3.8 GB)
16/09/07 19:10:25 INFO SparkContext: Created broadcast 977 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:25 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:25 INFO DAGScheduler: Registering RDD 1998 (repartition at SparkUtils.java:190)
16/09/07 19:10:25 INFO DAGScheduler: Got job 328 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:25 INFO DAGScheduler: Final stage: ResultStage 647(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 646)
16/09/07 19:10:25 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 646)
16/09/07 19:10:25 INFO DAGScheduler: Submitting ShuffleMapStage 646 (MapPartitionsRDD[1998] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:25 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18785921, maxMem=4042889625
16/09/07 19:10:25 INFO MemoryStore: Block broadcast_978 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:25 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18790385, maxMem=4042889625
16/09/07 19:10:25 INFO MemoryStore: Block broadcast_978_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:25 INFO BlockManagerInfo: Added broadcast_978_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:25 INFO SparkContext: Created broadcast 978 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:25 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 646 (MapPartitionsRDD[1998] at repartition at SparkUtils.java:190)
16/09/07 19:10:25 INFO TaskSchedulerImpl: Adding task set 646.0 with 8 tasks
16/09/07 19:10:25 INFO TaskSetManager: Starting task 0.0 in stage 646.0 (TID 5168, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 1.0 in stage 646.0 (TID 5169, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 2.0 in stage 646.0 (TID 5170, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 3.0 in stage 646.0 (TID 5171, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 4.0 in stage 646.0 (TID 5172, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 5.0 in stage 646.0 (TID 5173, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 6.0 in stage 646.0 (TID 5174, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 7.0 in stage 646.0 (TID 5175, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:25 INFO Executor: Running task 2.0 in stage 646.0 (TID 5170)
16/09/07 19:10:25 INFO Executor: Running task 6.0 in stage 646.0 (TID 5174)
16/09/07 19:10:25 INFO Executor: Running task 3.0 in stage 646.0 (TID 5171)
16/09/07 19:10:25 INFO Executor: Running task 0.0 in stage 646.0 (TID 5168)
16/09/07 19:10:25 INFO Executor: Running task 7.0 in stage 646.0 (TID 5175)
16/09/07 19:10:25 INFO Executor: Running task 1.0 in stage 646.0 (TID 5169)
16/09/07 19:10:25 INFO Executor: Running task 5.0 in stage 646.0 (TID 5173)
16/09/07 19:10:25 INFO Executor: Running task 4.0 in stage 646.0 (TID 5172)
16/09/07 19:10:25 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:25 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:25 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:25 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:25 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:25 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:25 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:25 INFO Executor: Finished task 7.0 in stage 646.0 (TID 5175). 2260 bytes result sent to driver
16/09/07 19:10:25 INFO Executor: Finished task 5.0 in stage 646.0 (TID 5173). 2260 bytes result sent to driver
16/09/07 19:10:25 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:25 INFO Executor: Finished task 4.0 in stage 646.0 (TID 5172). 2260 bytes result sent to driver
16/09/07 19:10:25 INFO TaskSetManager: Finished task 7.0 in stage 646.0 (TID 5175) in 2 ms on localhost (1/8)
16/09/07 19:10:25 INFO TaskSetManager: Finished task 5.0 in stage 646.0 (TID 5173) in 2 ms on localhost (2/8)
16/09/07 19:10:25 INFO TaskSetManager: Finished task 4.0 in stage 646.0 (TID 5172) in 2 ms on localhost (3/8)
16/09/07 19:10:25 INFO Executor: Finished task 3.0 in stage 646.0 (TID 5171). 2260 bytes result sent to driver
16/09/07 19:10:25 INFO Executor: Finished task 0.0 in stage 646.0 (TID 5168). 2260 bytes result sent to driver
16/09/07 19:10:25 INFO TaskSetManager: Finished task 3.0 in stage 646.0 (TID 5171) in 3 ms on localhost (4/8)
16/09/07 19:10:25 INFO Executor: Finished task 1.0 in stage 646.0 (TID 5169). 2260 bytes result sent to driver
16/09/07 19:10:25 INFO TaskSetManager: Finished task 0.0 in stage 646.0 (TID 5168) in 4 ms on localhost (5/8)
16/09/07 19:10:25 INFO TaskSetManager: Finished task 1.0 in stage 646.0 (TID 5169) in 3 ms on localhost (6/8)
16/09/07 19:10:25 INFO Executor: Finished task 2.0 in stage 646.0 (TID 5170). 2260 bytes result sent to driver
16/09/07 19:10:25 INFO Executor: Finished task 6.0 in stage 646.0 (TID 5174). 2260 bytes result sent to driver
16/09/07 19:10:25 INFO TaskSetManager: Finished task 2.0 in stage 646.0 (TID 5170) in 4 ms on localhost (7/8)
16/09/07 19:10:25 INFO TaskSetManager: Finished task 6.0 in stage 646.0 (TID 5174) in 4 ms on localhost (8/8)
16/09/07 19:10:25 INFO TaskSchedulerImpl: Removed TaskSet 646.0, whose tasks have all completed, from pool 
16/09/07 19:10:25 INFO DAGScheduler: ShuffleMapStage 646 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:25 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:25 INFO DAGScheduler: running: Set()
16/09/07 19:10:25 INFO DAGScheduler: waiting: Set(ResultStage 647)
16/09/07 19:10:25 INFO DAGScheduler: failed: Set()
16/09/07 19:10:25 INFO DAGScheduler: Missing parents for ResultStage 647: List()
16/09/07 19:10:25 INFO DAGScheduler: Submitting ResultStage 647 (MapPartitionsRDD[2002] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:25 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18792949, maxMem=4042889625
16/09/07 19:10:25 INFO MemoryStore: Block broadcast_979 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:25 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18798045, maxMem=4042889625
16/09/07 19:10:25 INFO MemoryStore: Block broadcast_979_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:25 INFO BlockManagerInfo: Added broadcast_979_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:25 INFO SparkContext: Created broadcast 979 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:25 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 647 (MapPartitionsRDD[2002] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:25 INFO TaskSchedulerImpl: Adding task set 647.0 with 8 tasks
16/09/07 19:10:25 INFO TaskSetManager: Starting task 0.0 in stage 647.0 (TID 5176, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 1.0 in stage 647.0 (TID 5177, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 2.0 in stage 647.0 (TID 5178, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 3.0 in stage 647.0 (TID 5179, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 4.0 in stage 647.0 (TID 5180, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 5.0 in stage 647.0 (TID 5181, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 6.0 in stage 647.0 (TID 5182, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:25 INFO TaskSetManager: Starting task 7.0 in stage 647.0 (TID 5183, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:25 INFO Executor: Running task 0.0 in stage 647.0 (TID 5176)
16/09/07 19:10:25 INFO Executor: Running task 6.0 in stage 647.0 (TID 5182)
16/09/07 19:10:25 INFO Executor: Running task 2.0 in stage 647.0 (TID 5178)
16/09/07 19:10:25 INFO Executor: Running task 5.0 in stage 647.0 (TID 5181)
16/09/07 19:10:25 INFO Executor: Running task 7.0 in stage 647.0 (TID 5183)
16/09/07 19:10:25 INFO Executor: Running task 1.0 in stage 647.0 (TID 5177)
16/09/07 19:10:25 INFO Executor: Running task 4.0 in stage 647.0 (TID 5180)
16/09/07 19:10:25 INFO Executor: Running task 3.0 in stage 647.0 (TID 5179)
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:25 INFO Executor: Finished task 4.0 in stage 647.0 (TID 5180). 2160 bytes result sent to driver
16/09/07 19:10:25 INFO Executor: Finished task 5.0 in stage 647.0 (TID 5181). 2160 bytes result sent to driver
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:25 INFO Executor: Finished task 3.0 in stage 647.0 (TID 5179). 2160 bytes result sent to driver
16/09/07 19:10:25 INFO Executor: Finished task 1.0 in stage 647.0 (TID 5177). 2160 bytes result sent to driver
16/09/07 19:10:25 INFO TaskSetManager: Finished task 4.0 in stage 647.0 (TID 5180) in 2 ms on localhost (1/8)
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:25 INFO TaskSetManager: Finished task 5.0 in stage 647.0 (TID 5181) in 2 ms on localhost (2/8)
16/09/07 19:10:25 INFO TaskSetManager: Finished task 3.0 in stage 647.0 (TID 5179) in 3 ms on localhost (3/8)
16/09/07 19:10:25 INFO TaskSetManager: Finished task 1.0 in stage 647.0 (TID 5177) in 3 ms on localhost (4/8)
16/09/07 19:10:25 INFO Executor: Finished task 2.0 in stage 647.0 (TID 5178). 2160 bytes result sent to driver
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:25 INFO TaskSetManager: Finished task 2.0 in stage 647.0 (TID 5178) in 3 ms on localhost (5/8)
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:25 INFO BlockManagerInfo: Removed broadcast_978_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:26 INFO Executor: Finished task 7.0 in stage 647.0 (TID 5183). 3485274 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 7.0 in stage 647.0 (TID 5183) in 853 ms on localhost (6/8)
16/09/07 19:10:26 INFO Executor: Finished task 6.0 in stage 647.0 (TID 5182). 3485274 bytes result sent to driver
16/09/07 19:10:26 INFO Executor: Finished task 0.0 in stage 647.0 (TID 5176). 3485274 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 6.0 in stage 647.0 (TID 5182) in 906 ms on localhost (7/8)
16/09/07 19:10:26 INFO TaskSetManager: Finished task 0.0 in stage 647.0 (TID 5176) in 908 ms on localhost (8/8)
16/09/07 19:10:26 INFO TaskSchedulerImpl: Removed TaskSet 647.0, whose tasks have all completed, from pool 
16/09/07 19:10:26 INFO DAGScheduler: ResultStage 647 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.909 s
16/09/07 19:10:26 INFO DAGScheduler: Job 328 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.916507 s
16/09/07 19:10:26 INFO ParameterAveragingTrainingMaster: Completed training of split 7 of 78
16/09/07 19:10:26 INFO ParameterAveragingTrainingMaster: Starting training of split 8 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:26 INFO BlockManagerInfo: Removed broadcast_979_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:26 INFO ContextCleaner: Cleaned accumulator 648
16/09/07 19:10:26 INFO ContextCleaner: Cleaned accumulator 647
16/09/07 19:10:26 INFO BlockManagerInfo: Removed broadcast_977_piece0 on localhost:41251 in memory (size: 270.1 KB, free: 3.8 GB)
16/09/07 19:10:26 INFO ContextCleaner: Cleaned shuffle 318
16/09/07 19:10:26 INFO MemoryStore: ensureFreeSpace(17680136) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:26 INFO MemoryStore: Block broadcast_980 stored as values in memory (estimated size 16.9 MB, free 3.7 GB)
16/09/07 19:10:26 INFO MemoryStore: ensureFreeSpace(276523) called with curMem=18223960, maxMem=4042889625
16/09/07 19:10:26 INFO MemoryStore: Block broadcast_980_piece0 stored as bytes in memory (estimated size 270.0 KB, free 3.7 GB)
16/09/07 19:10:26 INFO BlockManagerInfo: Added broadcast_980_piece0 in memory on localhost:41251 (size: 270.0 KB, free: 3.8 GB)
16/09/07 19:10:26 INFO SparkContext: Created broadcast 980 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:26 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:26 INFO DAGScheduler: Registering RDD 2003 (repartition at SparkUtils.java:190)
16/09/07 19:10:26 INFO DAGScheduler: Got job 329 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:26 INFO DAGScheduler: Final stage: ResultStage 649(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:26 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 648)
16/09/07 19:10:26 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 648)
16/09/07 19:10:26 INFO DAGScheduler: Submitting ShuffleMapStage 648 (MapPartitionsRDD[2003] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:26 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18500483, maxMem=4042889625
16/09/07 19:10:26 INFO MemoryStore: Block broadcast_981 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:26 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18504947, maxMem=4042889625
16/09/07 19:10:26 INFO MemoryStore: Block broadcast_981_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:26 INFO BlockManagerInfo: Added broadcast_981_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:26 INFO SparkContext: Created broadcast 981 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:26 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 648 (MapPartitionsRDD[2003] at repartition at SparkUtils.java:190)
16/09/07 19:10:26 INFO TaskSchedulerImpl: Adding task set 648.0 with 8 tasks
16/09/07 19:10:26 INFO TaskSetManager: Starting task 0.0 in stage 648.0 (TID 5184, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 1.0 in stage 648.0 (TID 5185, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 2.0 in stage 648.0 (TID 5186, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 3.0 in stage 648.0 (TID 5187, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 4.0 in stage 648.0 (TID 5188, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 5.0 in stage 648.0 (TID 5189, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 6.0 in stage 648.0 (TID 5190, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 7.0 in stage 648.0 (TID 5191, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:26 INFO Executor: Running task 3.0 in stage 648.0 (TID 5187)
16/09/07 19:10:26 INFO Executor: Running task 4.0 in stage 648.0 (TID 5188)
16/09/07 19:10:26 INFO Executor: Running task 6.0 in stage 648.0 (TID 5190)
16/09/07 19:10:26 INFO Executor: Running task 7.0 in stage 648.0 (TID 5191)
16/09/07 19:10:26 INFO Executor: Running task 5.0 in stage 648.0 (TID 5189)
16/09/07 19:10:26 INFO Executor: Running task 1.0 in stage 648.0 (TID 5185)
16/09/07 19:10:26 INFO Executor: Running task 2.0 in stage 648.0 (TID 5186)
16/09/07 19:10:26 INFO Executor: Running task 0.0 in stage 648.0 (TID 5184)
16/09/07 19:10:26 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:26 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:26 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:26 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:26 INFO Executor: Finished task 0.0 in stage 648.0 (TID 5184). 2260 bytes result sent to driver
16/09/07 19:10:26 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:26 INFO TaskSetManager: Finished task 0.0 in stage 648.0 (TID 5184) in 2 ms on localhost (1/8)
16/09/07 19:10:26 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:26 INFO Executor: Finished task 1.0 in stage 648.0 (TID 5185). 2260 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 1.0 in stage 648.0 (TID 5185) in 3 ms on localhost (2/8)
16/09/07 19:10:26 INFO Executor: Finished task 4.0 in stage 648.0 (TID 5188). 2260 bytes result sent to driver
16/09/07 19:10:26 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:26 INFO Executor: Finished task 6.0 in stage 648.0 (TID 5190). 2260 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 4.0 in stage 648.0 (TID 5188) in 3 ms on localhost (3/8)
16/09/07 19:10:26 INFO TaskSetManager: Finished task 6.0 in stage 648.0 (TID 5190) in 2 ms on localhost (4/8)
16/09/07 19:10:26 INFO Executor: Finished task 3.0 in stage 648.0 (TID 5187). 2260 bytes result sent to driver
16/09/07 19:10:26 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:26 INFO Executor: Finished task 7.0 in stage 648.0 (TID 5191). 2260 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 3.0 in stage 648.0 (TID 5187) in 3 ms on localhost (5/8)
16/09/07 19:10:26 INFO TaskSetManager: Finished task 7.0 in stage 648.0 (TID 5191) in 3 ms on localhost (6/8)
16/09/07 19:10:26 INFO Executor: Finished task 5.0 in stage 648.0 (TID 5189). 2260 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 5.0 in stage 648.0 (TID 5189) in 3 ms on localhost (7/8)
16/09/07 19:10:26 INFO Executor: Finished task 2.0 in stage 648.0 (TID 5186). 2260 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 2.0 in stage 648.0 (TID 5186) in 4 ms on localhost (8/8)
16/09/07 19:10:26 INFO TaskSchedulerImpl: Removed TaskSet 648.0, whose tasks have all completed, from pool 
16/09/07 19:10:26 INFO DAGScheduler: ShuffleMapStage 648 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:26 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:26 INFO DAGScheduler: running: Set()
16/09/07 19:10:26 INFO DAGScheduler: waiting: Set(ResultStage 649)
16/09/07 19:10:26 INFO DAGScheduler: failed: Set()
16/09/07 19:10:26 INFO DAGScheduler: Missing parents for ResultStage 649: List()
16/09/07 19:10:26 INFO DAGScheduler: Submitting ResultStage 649 (MapPartitionsRDD[2007] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:26 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18507511, maxMem=4042889625
16/09/07 19:10:26 INFO MemoryStore: Block broadcast_982 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:26 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18512607, maxMem=4042889625
16/09/07 19:10:26 INFO MemoryStore: Block broadcast_982_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:26 INFO BlockManagerInfo: Added broadcast_982_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:26 INFO SparkContext: Created broadcast 982 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:26 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 649 (MapPartitionsRDD[2007] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:26 INFO TaskSchedulerImpl: Adding task set 649.0 with 8 tasks
16/09/07 19:10:26 INFO TaskSetManager: Starting task 0.0 in stage 649.0 (TID 5192, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 1.0 in stage 649.0 (TID 5193, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 2.0 in stage 649.0 (TID 5194, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 3.0 in stage 649.0 (TID 5195, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 4.0 in stage 649.0 (TID 5196, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 5.0 in stage 649.0 (TID 5197, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 6.0 in stage 649.0 (TID 5198, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:26 INFO TaskSetManager: Starting task 7.0 in stage 649.0 (TID 5199, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:26 INFO Executor: Running task 0.0 in stage 649.0 (TID 5192)
16/09/07 19:10:26 INFO Executor: Running task 4.0 in stage 649.0 (TID 5196)
16/09/07 19:10:26 INFO Executor: Running task 6.0 in stage 649.0 (TID 5198)
16/09/07 19:10:26 INFO Executor: Running task 1.0 in stage 649.0 (TID 5193)
16/09/07 19:10:26 INFO Executor: Running task 5.0 in stage 649.0 (TID 5197)
16/09/07 19:10:26 INFO Executor: Running task 7.0 in stage 649.0 (TID 5199)
16/09/07 19:10:26 INFO Executor: Running task 3.0 in stage 649.0 (TID 5195)
16/09/07 19:10:26 INFO Executor: Running task 2.0 in stage 649.0 (TID 5194)
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:26 INFO Executor: Finished task 4.0 in stage 649.0 (TID 5196). 2160 bytes result sent to driver
16/09/07 19:10:26 INFO Executor: Finished task 2.0 in stage 649.0 (TID 5194). 2160 bytes result sent to driver
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:26 INFO TaskSetManager: Finished task 4.0 in stage 649.0 (TID 5196) in 1 ms on localhost (1/8)
16/09/07 19:10:26 INFO TaskSetManager: Finished task 2.0 in stage 649.0 (TID 5194) in 1 ms on localhost (2/8)
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:26 INFO Executor: Finished task 1.0 in stage 649.0 (TID 5193). 2160 bytes result sent to driver
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:26 INFO TaskSetManager: Finished task 1.0 in stage 649.0 (TID 5193) in 2 ms on localhost (3/8)
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:26 INFO Executor: Finished task 3.0 in stage 649.0 (TID 5195). 2160 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 3.0 in stage 649.0 (TID 5195) in 2 ms on localhost (4/8)
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:26 INFO Executor: Finished task 5.0 in stage 649.0 (TID 5197). 2160 bytes result sent to driver
16/09/07 19:10:26 INFO TaskSetManager: Finished task 5.0 in stage 649.0 (TID 5197) in 3 ms on localhost (5/8)
16/09/07 19:10:27 INFO BlockManagerInfo: Removed broadcast_981_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:27 INFO Executor: Finished task 0.0 in stage 649.0 (TID 5192). 3485274 bytes result sent to driver
16/09/07 19:10:27 INFO Executor: Finished task 6.0 in stage 649.0 (TID 5198). 3485274 bytes result sent to driver
16/09/07 19:10:27 INFO Executor: Finished task 7.0 in stage 649.0 (TID 5199). 3485274 bytes result sent to driver
16/09/07 19:10:27 INFO TaskSetManager: Finished task 0.0 in stage 649.0 (TID 5192) in 908 ms on localhost (6/8)
16/09/07 19:10:27 INFO TaskSetManager: Finished task 6.0 in stage 649.0 (TID 5198) in 910 ms on localhost (7/8)
16/09/07 19:10:27 INFO TaskSetManager: Finished task 7.0 in stage 649.0 (TID 5199) in 915 ms on localhost (8/8)
16/09/07 19:10:27 INFO TaskSchedulerImpl: Removed TaskSet 649.0, whose tasks have all completed, from pool 
16/09/07 19:10:27 INFO DAGScheduler: ResultStage 649 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.915 s
16/09/07 19:10:27 INFO DAGScheduler: Job 329 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.922926 s
16/09/07 19:10:27 INFO ParameterAveragingTrainingMaster: Completed training of split 8 of 78
16/09/07 19:10:27 INFO ParameterAveragingTrainingMaster: Starting training of split 9 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:27 INFO MemoryStore: ensureFreeSpace(17995992) called with curMem=18508322, maxMem=4042889625
16/09/07 19:10:27 INFO MemoryStore: Block broadcast_983 stored as values in memory (estimated size 17.2 MB, free 3.7 GB)
16/09/07 19:10:27 INFO BlockManagerInfo: Removed broadcast_982_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:27 INFO ContextCleaner: Cleaned accumulator 650
16/09/07 19:10:27 INFO ContextCleaner: Cleaned accumulator 649
16/09/07 19:10:27 INFO BlockManagerInfo: Removed broadcast_980_piece0 on localhost:41251 in memory (size: 270.0 KB, free: 3.8 GB)
16/09/07 19:10:27 INFO ContextCleaner: Cleaned shuffle 319
16/09/07 19:10:27 INFO MemoryStore: ensureFreeSpace(276408) called with curMem=18539816, maxMem=4042889625
16/09/07 19:10:27 INFO MemoryStore: Block broadcast_983_piece0 stored as bytes in memory (estimated size 269.9 KB, free 3.7 GB)
16/09/07 19:10:27 INFO BlockManagerInfo: Added broadcast_983_piece0 in memory on localhost:41251 (size: 269.9 KB, free: 3.8 GB)
16/09/07 19:10:27 INFO SparkContext: Created broadcast 983 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:27 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:27 INFO DAGScheduler: Registering RDD 2008 (repartition at SparkUtils.java:190)
16/09/07 19:10:27 INFO DAGScheduler: Got job 330 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:27 INFO DAGScheduler: Final stage: ResultStage 651(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:27 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 650)
16/09/07 19:10:27 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 650)
16/09/07 19:10:27 INFO DAGScheduler: Submitting ShuffleMapStage 650 (MapPartitionsRDD[2008] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:27 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18816224, maxMem=4042889625
16/09/07 19:10:27 INFO MemoryStore: Block broadcast_984 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:27 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=18820688, maxMem=4042889625
16/09/07 19:10:27 INFO MemoryStore: Block broadcast_984_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:27 INFO BlockManagerInfo: Added broadcast_984_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:27 INFO SparkContext: Created broadcast 984 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:27 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 650 (MapPartitionsRDD[2008] at repartition at SparkUtils.java:190)
16/09/07 19:10:27 INFO TaskSchedulerImpl: Adding task set 650.0 with 8 tasks
16/09/07 19:10:27 INFO TaskSetManager: Starting task 0.0 in stage 650.0 (TID 5200, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 1.0 in stage 650.0 (TID 5201, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 2.0 in stage 650.0 (TID 5202, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 3.0 in stage 650.0 (TID 5203, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 4.0 in stage 650.0 (TID 5204, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 5.0 in stage 650.0 (TID 5205, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 6.0 in stage 650.0 (TID 5206, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 7.0 in stage 650.0 (TID 5207, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:27 INFO Executor: Running task 1.0 in stage 650.0 (TID 5201)
16/09/07 19:10:27 INFO Executor: Running task 2.0 in stage 650.0 (TID 5202)
16/09/07 19:10:27 INFO Executor: Running task 4.0 in stage 650.0 (TID 5204)
16/09/07 19:10:27 INFO Executor: Running task 5.0 in stage 650.0 (TID 5205)
16/09/07 19:10:27 INFO Executor: Running task 3.0 in stage 650.0 (TID 5203)
16/09/07 19:10:27 INFO Executor: Running task 0.0 in stage 650.0 (TID 5200)
16/09/07 19:10:27 INFO Executor: Running task 6.0 in stage 650.0 (TID 5206)
16/09/07 19:10:27 INFO Executor: Running task 7.0 in stage 650.0 (TID 5207)
16/09/07 19:10:27 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:27 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:27 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:27 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:27 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:27 INFO Executor: Finished task 2.0 in stage 650.0 (TID 5202). 2260 bytes result sent to driver
16/09/07 19:10:27 INFO TaskSetManager: Finished task 2.0 in stage 650.0 (TID 5202) in 2 ms on localhost (1/8)
16/09/07 19:10:27 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:27 INFO Executor: Finished task 5.0 in stage 650.0 (TID 5205). 2260 bytes result sent to driver
16/09/07 19:10:27 INFO Executor: Finished task 7.0 in stage 650.0 (TID 5207). 2260 bytes result sent to driver
16/09/07 19:10:27 INFO Executor: Finished task 3.0 in stage 650.0 (TID 5203). 2260 bytes result sent to driver
16/09/07 19:10:27 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:27 INFO Executor: Finished task 6.0 in stage 650.0 (TID 5206). 2260 bytes result sent to driver
16/09/07 19:10:27 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:27 INFO TaskSetManager: Finished task 7.0 in stage 650.0 (TID 5207) in 3 ms on localhost (2/8)
16/09/07 19:10:27 INFO TaskSetManager: Finished task 6.0 in stage 650.0 (TID 5206) in 3 ms on localhost (3/8)
16/09/07 19:10:27 INFO TaskSetManager: Finished task 3.0 in stage 650.0 (TID 5203) in 3 ms on localhost (4/8)
16/09/07 19:10:27 INFO TaskSetManager: Finished task 5.0 in stage 650.0 (TID 5205) in 3 ms on localhost (5/8)
16/09/07 19:10:27 INFO Executor: Finished task 1.0 in stage 650.0 (TID 5201). 2260 bytes result sent to driver
16/09/07 19:10:27 INFO TaskSetManager: Finished task 1.0 in stage 650.0 (TID 5201) in 3 ms on localhost (6/8)
16/09/07 19:10:27 INFO Executor: Finished task 0.0 in stage 650.0 (TID 5200). 2260 bytes result sent to driver
16/09/07 19:10:27 INFO Executor: Finished task 4.0 in stage 650.0 (TID 5204). 2260 bytes result sent to driver
16/09/07 19:10:27 INFO TaskSetManager: Finished task 0.0 in stage 650.0 (TID 5200) in 3 ms on localhost (7/8)
16/09/07 19:10:27 INFO TaskSetManager: Finished task 4.0 in stage 650.0 (TID 5204) in 4 ms on localhost (8/8)
16/09/07 19:10:27 INFO TaskSchedulerImpl: Removed TaskSet 650.0, whose tasks have all completed, from pool 
16/09/07 19:10:27 INFO DAGScheduler: ShuffleMapStage 650 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:27 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:27 INFO DAGScheduler: running: Set()
16/09/07 19:10:27 INFO DAGScheduler: waiting: Set(ResultStage 651)
16/09/07 19:10:27 INFO DAGScheduler: failed: Set()
16/09/07 19:10:27 INFO DAGScheduler: Missing parents for ResultStage 651: List()
16/09/07 19:10:27 INFO DAGScheduler: Submitting ResultStage 651 (MapPartitionsRDD[2012] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:27 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18823250, maxMem=4042889625
16/09/07 19:10:27 INFO MemoryStore: Block broadcast_985 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:27 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18828346, maxMem=4042889625
16/09/07 19:10:27 INFO MemoryStore: Block broadcast_985_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:27 INFO BlockManagerInfo: Added broadcast_985_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:27 INFO SparkContext: Created broadcast 985 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:27 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 651 (MapPartitionsRDD[2012] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:27 INFO TaskSchedulerImpl: Adding task set 651.0 with 8 tasks
16/09/07 19:10:27 INFO TaskSetManager: Starting task 0.0 in stage 651.0 (TID 5208, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 1.0 in stage 651.0 (TID 5209, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 2.0 in stage 651.0 (TID 5210, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 3.0 in stage 651.0 (TID 5211, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 4.0 in stage 651.0 (TID 5212, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 5.0 in stage 651.0 (TID 5213, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 6.0 in stage 651.0 (TID 5214, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:27 INFO TaskSetManager: Starting task 7.0 in stage 651.0 (TID 5215, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:27 INFO Executor: Running task 0.0 in stage 651.0 (TID 5208)
16/09/07 19:10:27 INFO Executor: Running task 5.0 in stage 651.0 (TID 5213)
16/09/07 19:10:27 INFO Executor: Running task 1.0 in stage 651.0 (TID 5209)
16/09/07 19:10:27 INFO Executor: Running task 6.0 in stage 651.0 (TID 5214)
16/09/07 19:10:27 INFO Executor: Running task 2.0 in stage 651.0 (TID 5210)
16/09/07 19:10:27 INFO Executor: Running task 7.0 in stage 651.0 (TID 5215)
16/09/07 19:10:27 INFO Executor: Running task 4.0 in stage 651.0 (TID 5212)
16/09/07 19:10:27 INFO Executor: Running task 3.0 in stage 651.0 (TID 5211)
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:27 INFO Executor: Finished task 4.0 in stage 651.0 (TID 5212). 2160 bytes result sent to driver
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:27 INFO Executor: Finished task 5.0 in stage 651.0 (TID 5213). 2160 bytes result sent to driver
16/09/07 19:10:27 INFO Executor: Finished task 3.0 in stage 651.0 (TID 5211). 2160 bytes result sent to driver
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:27 INFO TaskSetManager: Finished task 4.0 in stage 651.0 (TID 5212) in 2 ms on localhost (1/8)
16/09/07 19:10:27 INFO TaskSetManager: Finished task 5.0 in stage 651.0 (TID 5213) in 2 ms on localhost (2/8)
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:27 INFO Executor: Finished task 2.0 in stage 651.0 (TID 5210). 2160 bytes result sent to driver
16/09/07 19:10:27 INFO TaskSetManager: Finished task 3.0 in stage 651.0 (TID 5211) in 2 ms on localhost (3/8)
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:27 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:27 INFO TaskSetManager: Finished task 2.0 in stage 651.0 (TID 5210) in 2 ms on localhost (4/8)
16/09/07 19:10:27 INFO Executor: Finished task 1.0 in stage 651.0 (TID 5209). 2160 bytes result sent to driver
16/09/07 19:10:27 INFO TaskSetManager: Finished task 1.0 in stage 651.0 (TID 5209) in 3 ms on localhost (5/8)
16/09/07 19:10:28 INFO BlockManagerInfo: Removed broadcast_984_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:28 INFO Executor: Finished task 0.0 in stage 651.0 (TID 5208). 3485274 bytes result sent to driver
16/09/07 19:10:28 INFO TaskSetManager: Finished task 0.0 in stage 651.0 (TID 5208) in 887 ms on localhost (6/8)
16/09/07 19:10:28 INFO Executor: Finished task 6.0 in stage 651.0 (TID 5214). 3485274 bytes result sent to driver
16/09/07 19:10:28 INFO Executor: Finished task 7.0 in stage 651.0 (TID 5215). 3485274 bytes result sent to driver
16/09/07 19:10:28 INFO TaskSetManager: Finished task 6.0 in stage 651.0 (TID 5214) in 931 ms on localhost (7/8)
16/09/07 19:10:28 INFO TaskSetManager: Finished task 7.0 in stage 651.0 (TID 5215) in 944 ms on localhost (8/8)
16/09/07 19:10:28 INFO TaskSchedulerImpl: Removed TaskSet 651.0, whose tasks have all completed, from pool 
16/09/07 19:10:28 INFO DAGScheduler: ResultStage 651 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.944 s
16/09/07 19:10:28 INFO DAGScheduler: Job 330 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.951324 s
16/09/07 19:10:28 INFO ParameterAveragingTrainingMaster: Completed training of split 9 of 78
16/09/07 19:10:28 INFO ParameterAveragingTrainingMaster: Starting training of split 10 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:29 INFO BlockManagerInfo: Removed broadcast_985_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:29 INFO ContextCleaner: Cleaned accumulator 652
16/09/07 19:10:29 INFO ContextCleaner: Cleaned accumulator 651
16/09/07 19:10:29 INFO BlockManagerInfo: Removed broadcast_983_piece0 on localhost:41251 in memory (size: 269.9 KB, free: 3.8 GB)
16/09/07 19:10:29 INFO ContextCleaner: Cleaned shuffle 320
16/09/07 19:10:29 INFO MemoryStore: ensureFreeSpace(17701544) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:29 INFO MemoryStore: Block broadcast_986 stored as values in memory (estimated size 16.9 MB, free 3.7 GB)
16/09/07 19:10:29 INFO MemoryStore: ensureFreeSpace(276934) called with curMem=18245368, maxMem=4042889625
16/09/07 19:10:29 INFO MemoryStore: Block broadcast_986_piece0 stored as bytes in memory (estimated size 270.4 KB, free 3.7 GB)
16/09/07 19:10:29 INFO BlockManagerInfo: Added broadcast_986_piece0 in memory on localhost:41251 (size: 270.4 KB, free: 3.8 GB)
16/09/07 19:10:29 INFO SparkContext: Created broadcast 986 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:29 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:29 INFO DAGScheduler: Registering RDD 2013 (repartition at SparkUtils.java:190)
16/09/07 19:10:29 INFO DAGScheduler: Got job 331 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:29 INFO DAGScheduler: Final stage: ResultStage 653(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:29 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 652)
16/09/07 19:10:29 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 652)
16/09/07 19:10:29 INFO DAGScheduler: Submitting ShuffleMapStage 652 (MapPartitionsRDD[2013] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:29 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18522302, maxMem=4042889625
16/09/07 19:10:29 INFO MemoryStore: Block broadcast_987 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:29 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18526766, maxMem=4042889625
16/09/07 19:10:29 INFO MemoryStore: Block broadcast_987_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:29 INFO BlockManagerInfo: Added broadcast_987_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:29 INFO SparkContext: Created broadcast 987 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:29 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 652 (MapPartitionsRDD[2013] at repartition at SparkUtils.java:190)
16/09/07 19:10:29 INFO TaskSchedulerImpl: Adding task set 652.0 with 8 tasks
16/09/07 19:10:29 INFO TaskSetManager: Starting task 0.0 in stage 652.0 (TID 5216, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 1.0 in stage 652.0 (TID 5217, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 2.0 in stage 652.0 (TID 5218, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 3.0 in stage 652.0 (TID 5219, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 4.0 in stage 652.0 (TID 5220, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 5.0 in stage 652.0 (TID 5221, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 6.0 in stage 652.0 (TID 5222, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 7.0 in stage 652.0 (TID 5223, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:29 INFO Executor: Running task 1.0 in stage 652.0 (TID 5217)
16/09/07 19:10:29 INFO Executor: Running task 6.0 in stage 652.0 (TID 5222)
16/09/07 19:10:29 INFO Executor: Running task 2.0 in stage 652.0 (TID 5218)
16/09/07 19:10:29 INFO Executor: Running task 4.0 in stage 652.0 (TID 5220)
16/09/07 19:10:29 INFO Executor: Running task 3.0 in stage 652.0 (TID 5219)
16/09/07 19:10:29 INFO Executor: Running task 0.0 in stage 652.0 (TID 5216)
16/09/07 19:10:29 INFO Executor: Running task 5.0 in stage 652.0 (TID 5221)
16/09/07 19:10:29 INFO Executor: Running task 7.0 in stage 652.0 (TID 5223)
16/09/07 19:10:29 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:29 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:29 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:29 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:29 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:29 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:29 INFO Executor: Finished task 1.0 in stage 652.0 (TID 5217). 2260 bytes result sent to driver
16/09/07 19:10:29 INFO Executor: Finished task 3.0 in stage 652.0 (TID 5219). 2260 bytes result sent to driver
16/09/07 19:10:29 INFO Executor: Finished task 0.0 in stage 652.0 (TID 5216). 2260 bytes result sent to driver
16/09/07 19:10:29 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:29 INFO TaskSetManager: Finished task 1.0 in stage 652.0 (TID 5217) in 2 ms on localhost (1/8)
16/09/07 19:10:29 INFO TaskSetManager: Finished task 3.0 in stage 652.0 (TID 5219) in 2 ms on localhost (2/8)
16/09/07 19:10:29 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:29 INFO TaskSetManager: Finished task 0.0 in stage 652.0 (TID 5216) in 3 ms on localhost (3/8)
16/09/07 19:10:29 INFO Executor: Finished task 5.0 in stage 652.0 (TID 5221). 2260 bytes result sent to driver
16/09/07 19:10:29 INFO TaskSetManager: Finished task 5.0 in stage 652.0 (TID 5221) in 3 ms on localhost (4/8)
16/09/07 19:10:29 INFO Executor: Finished task 2.0 in stage 652.0 (TID 5218). 2260 bytes result sent to driver
16/09/07 19:10:29 INFO TaskSetManager: Finished task 2.0 in stage 652.0 (TID 5218) in 3 ms on localhost (5/8)
16/09/07 19:10:29 INFO Executor: Finished task 4.0 in stage 652.0 (TID 5220). 2260 bytes result sent to driver
16/09/07 19:10:29 INFO TaskSetManager: Finished task 4.0 in stage 652.0 (TID 5220) in 3 ms on localhost (6/8)
16/09/07 19:10:29 INFO Executor: Finished task 6.0 in stage 652.0 (TID 5222). 2260 bytes result sent to driver
16/09/07 19:10:29 INFO TaskSetManager: Finished task 6.0 in stage 652.0 (TID 5222) in 3 ms on localhost (7/8)
16/09/07 19:10:29 INFO Executor: Finished task 7.0 in stage 652.0 (TID 5223). 2260 bytes result sent to driver
16/09/07 19:10:29 INFO TaskSetManager: Finished task 7.0 in stage 652.0 (TID 5223) in 3 ms on localhost (8/8)
16/09/07 19:10:29 INFO TaskSchedulerImpl: Removed TaskSet 652.0, whose tasks have all completed, from pool 
16/09/07 19:10:29 INFO DAGScheduler: ShuffleMapStage 652 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:29 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:29 INFO DAGScheduler: running: Set()
16/09/07 19:10:29 INFO DAGScheduler: waiting: Set(ResultStage 653)
16/09/07 19:10:29 INFO DAGScheduler: failed: Set()
16/09/07 19:10:29 INFO DAGScheduler: Missing parents for ResultStage 653: List()
16/09/07 19:10:29 INFO DAGScheduler: Submitting ResultStage 653 (MapPartitionsRDD[2017] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:29 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18529330, maxMem=4042889625
16/09/07 19:10:29 INFO MemoryStore: Block broadcast_988 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:29 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18534426, maxMem=4042889625
16/09/07 19:10:29 INFO MemoryStore: Block broadcast_988_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:29 INFO BlockManagerInfo: Added broadcast_988_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:29 INFO SparkContext: Created broadcast 988 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:29 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 653 (MapPartitionsRDD[2017] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:29 INFO TaskSchedulerImpl: Adding task set 653.0 with 8 tasks
16/09/07 19:10:29 INFO TaskSetManager: Starting task 0.0 in stage 653.0 (TID 5224, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 1.0 in stage 653.0 (TID 5225, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 2.0 in stage 653.0 (TID 5226, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 3.0 in stage 653.0 (TID 5227, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 4.0 in stage 653.0 (TID 5228, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 5.0 in stage 653.0 (TID 5229, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 6.0 in stage 653.0 (TID 5230, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:29 INFO TaskSetManager: Starting task 7.0 in stage 653.0 (TID 5231, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:29 INFO Executor: Running task 0.0 in stage 653.0 (TID 5224)
16/09/07 19:10:29 INFO Executor: Running task 3.0 in stage 653.0 (TID 5227)
16/09/07 19:10:29 INFO Executor: Running task 7.0 in stage 653.0 (TID 5231)
16/09/07 19:10:29 INFO Executor: Running task 1.0 in stage 653.0 (TID 5225)
16/09/07 19:10:29 INFO Executor: Running task 2.0 in stage 653.0 (TID 5226)
16/09/07 19:10:29 INFO Executor: Running task 6.0 in stage 653.0 (TID 5230)
16/09/07 19:10:29 INFO Executor: Running task 4.0 in stage 653.0 (TID 5228)
16/09/07 19:10:29 INFO Executor: Running task 5.0 in stage 653.0 (TID 5229)
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:29 INFO Executor: Finished task 5.0 in stage 653.0 (TID 5229). 2160 bytes result sent to driver
16/09/07 19:10:29 INFO Executor: Finished task 4.0 in stage 653.0 (TID 5228). 2160 bytes result sent to driver
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:29 INFO TaskSetManager: Finished task 5.0 in stage 653.0 (TID 5229) in 2 ms on localhost (1/8)
16/09/07 19:10:29 INFO TaskSetManager: Finished task 4.0 in stage 653.0 (TID 5228) in 2 ms on localhost (2/8)
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:29 INFO Executor: Finished task 1.0 in stage 653.0 (TID 5225). 2160 bytes result sent to driver
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:29 INFO TaskSetManager: Finished task 1.0 in stage 653.0 (TID 5225) in 2 ms on localhost (3/8)
16/09/07 19:10:29 INFO Executor: Finished task 3.0 in stage 653.0 (TID 5227). 2160 bytes result sent to driver
16/09/07 19:10:29 INFO Executor: Finished task 2.0 in stage 653.0 (TID 5226). 2160 bytes result sent to driver
16/09/07 19:10:29 INFO TaskSetManager: Finished task 3.0 in stage 653.0 (TID 5227) in 2 ms on localhost (4/8)
16/09/07 19:10:29 INFO TaskSetManager: Finished task 2.0 in stage 653.0 (TID 5226) in 3 ms on localhost (5/8)
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:29 INFO BlockManagerInfo: Removed broadcast_987_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:29 INFO Executor: Finished task 6.0 in stage 653.0 (TID 5230). 3485274 bytes result sent to driver
16/09/07 19:10:29 INFO Executor: Finished task 0.0 in stage 653.0 (TID 5224). 3485274 bytes result sent to driver
16/09/07 19:10:29 INFO Executor: Finished task 7.0 in stage 653.0 (TID 5231). 3485274 bytes result sent to driver
16/09/07 19:10:30 INFO TaskSetManager: Finished task 6.0 in stage 653.0 (TID 5230) in 880 ms on localhost (6/8)
16/09/07 19:10:30 INFO TaskSetManager: Finished task 7.0 in stage 653.0 (TID 5231) in 883 ms on localhost (7/8)
16/09/07 19:10:30 INFO TaskSetManager: Finished task 0.0 in stage 653.0 (TID 5224) in 883 ms on localhost (8/8)
16/09/07 19:10:30 INFO TaskSchedulerImpl: Removed TaskSet 653.0, whose tasks have all completed, from pool 
16/09/07 19:10:30 INFO DAGScheduler: ResultStage 653 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.883 s
16/09/07 19:10:30 INFO DAGScheduler: Job 331 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.891400 s
16/09/07 19:10:30 INFO ParameterAveragingTrainingMaster: Completed training of split 10 of 78
16/09/07 19:10:30 INFO ParameterAveragingTrainingMaster: Starting training of split 11 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:30 INFO MemoryStore: ensureFreeSpace(18013920) called with curMem=18530141, maxMem=4042889625
16/09/07 19:10:30 INFO MemoryStore: Block broadcast_989 stored as values in memory (estimated size 17.2 MB, free 3.7 GB)
16/09/07 19:10:30 INFO BlockManagerInfo: Removed broadcast_988_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:30 INFO ContextCleaner: Cleaned accumulator 654
16/09/07 19:10:30 INFO ContextCleaner: Cleaned accumulator 653
16/09/07 19:10:30 INFO BlockManagerInfo: Removed broadcast_986_piece0 on localhost:41251 in memory (size: 270.4 KB, free: 3.8 GB)
16/09/07 19:10:30 INFO ContextCleaner: Cleaned shuffle 321
16/09/07 19:10:30 INFO MemoryStore: ensureFreeSpace(276340) called with curMem=18557744, maxMem=4042889625
16/09/07 19:10:30 INFO MemoryStore: Block broadcast_989_piece0 stored as bytes in memory (estimated size 269.9 KB, free 3.7 GB)
16/09/07 19:10:30 INFO BlockManagerInfo: Added broadcast_989_piece0 in memory on localhost:41251 (size: 269.9 KB, free: 3.8 GB)
16/09/07 19:10:30 INFO SparkContext: Created broadcast 989 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:30 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:30 INFO DAGScheduler: Registering RDD 2018 (repartition at SparkUtils.java:190)
16/09/07 19:10:30 INFO DAGScheduler: Got job 332 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:30 INFO DAGScheduler: Final stage: ResultStage 655(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:30 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 654)
16/09/07 19:10:30 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 654)
16/09/07 19:10:30 INFO DAGScheduler: Submitting ShuffleMapStage 654 (MapPartitionsRDD[2018] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:30 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18834084, maxMem=4042889625
16/09/07 19:10:30 INFO MemoryStore: Block broadcast_990 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:30 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18838548, maxMem=4042889625
16/09/07 19:10:30 INFO MemoryStore: Block broadcast_990_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:30 INFO BlockManagerInfo: Added broadcast_990_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:30 INFO SparkContext: Created broadcast 990 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:30 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 654 (MapPartitionsRDD[2018] at repartition at SparkUtils.java:190)
16/09/07 19:10:30 INFO TaskSchedulerImpl: Adding task set 654.0 with 8 tasks
16/09/07 19:10:30 INFO TaskSetManager: Starting task 0.0 in stage 654.0 (TID 5232, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 1.0 in stage 654.0 (TID 5233, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 2.0 in stage 654.0 (TID 5234, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 3.0 in stage 654.0 (TID 5235, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 4.0 in stage 654.0 (TID 5236, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 5.0 in stage 654.0 (TID 5237, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 6.0 in stage 654.0 (TID 5238, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 7.0 in stage 654.0 (TID 5239, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:30 INFO Executor: Running task 4.0 in stage 654.0 (TID 5236)
16/09/07 19:10:30 INFO Executor: Running task 1.0 in stage 654.0 (TID 5233)
16/09/07 19:10:30 INFO Executor: Running task 6.0 in stage 654.0 (TID 5238)
16/09/07 19:10:30 INFO Executor: Running task 5.0 in stage 654.0 (TID 5237)
16/09/07 19:10:30 INFO Executor: Running task 3.0 in stage 654.0 (TID 5235)
16/09/07 19:10:30 INFO Executor: Running task 2.0 in stage 654.0 (TID 5234)
16/09/07 19:10:30 INFO Executor: Running task 0.0 in stage 654.0 (TID 5232)
16/09/07 19:10:30 INFO Executor: Running task 7.0 in stage 654.0 (TID 5239)
16/09/07 19:10:30 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:30 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:30 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:30 INFO Executor: Finished task 4.0 in stage 654.0 (TID 5236). 2260 bytes result sent to driver
16/09/07 19:10:30 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:30 INFO TaskSetManager: Finished task 4.0 in stage 654.0 (TID 5236) in 2 ms on localhost (1/8)
16/09/07 19:10:30 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:30 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:30 INFO Executor: Finished task 7.0 in stage 654.0 (TID 5239). 2260 bytes result sent to driver
16/09/07 19:10:30 INFO Executor: Finished task 2.0 in stage 654.0 (TID 5234). 2260 bytes result sent to driver
16/09/07 19:10:30 INFO TaskSetManager: Finished task 7.0 in stage 654.0 (TID 5239) in 2 ms on localhost (2/8)
16/09/07 19:10:30 INFO Executor: Finished task 0.0 in stage 654.0 (TID 5232). 2260 bytes result sent to driver
16/09/07 19:10:30 INFO TaskSetManager: Finished task 2.0 in stage 654.0 (TID 5234) in 3 ms on localhost (3/8)
16/09/07 19:10:30 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:30 INFO TaskSetManager: Finished task 0.0 in stage 654.0 (TID 5232) in 3 ms on localhost (4/8)
16/09/07 19:10:30 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:30 INFO Executor: Finished task 3.0 in stage 654.0 (TID 5235). 2260 bytes result sent to driver
16/09/07 19:10:30 INFO Executor: Finished task 6.0 in stage 654.0 (TID 5238). 2260 bytes result sent to driver
16/09/07 19:10:30 INFO Executor: Finished task 5.0 in stage 654.0 (TID 5237). 2260 bytes result sent to driver
16/09/07 19:10:30 INFO TaskSetManager: Finished task 3.0 in stage 654.0 (TID 5235) in 4 ms on localhost (5/8)
16/09/07 19:10:30 INFO TaskSetManager: Finished task 6.0 in stage 654.0 (TID 5238) in 3 ms on localhost (6/8)
16/09/07 19:10:30 INFO Executor: Finished task 1.0 in stage 654.0 (TID 5233). 2260 bytes result sent to driver
16/09/07 19:10:30 INFO TaskSetManager: Finished task 5.0 in stage 654.0 (TID 5237) in 3 ms on localhost (7/8)
16/09/07 19:10:30 INFO TaskSetManager: Finished task 1.0 in stage 654.0 (TID 5233) in 5 ms on localhost (8/8)
16/09/07 19:10:30 INFO DAGScheduler: ShuffleMapStage 654 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:30 INFO TaskSchedulerImpl: Removed TaskSet 654.0, whose tasks have all completed, from pool 
16/09/07 19:10:30 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:30 INFO DAGScheduler: running: Set()
16/09/07 19:10:30 INFO DAGScheduler: waiting: Set(ResultStage 655)
16/09/07 19:10:30 INFO DAGScheduler: failed: Set()
16/09/07 19:10:30 INFO DAGScheduler: Missing parents for ResultStage 655: List()
16/09/07 19:10:30 INFO DAGScheduler: Submitting ResultStage 655 (MapPartitionsRDD[2022] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:30 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18841112, maxMem=4042889625
16/09/07 19:10:30 INFO MemoryStore: Block broadcast_991 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:30 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18846208, maxMem=4042889625
16/09/07 19:10:30 INFO MemoryStore: Block broadcast_991_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:30 INFO BlockManagerInfo: Added broadcast_991_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:30 INFO SparkContext: Created broadcast 991 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:30 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 655 (MapPartitionsRDD[2022] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:30 INFO TaskSchedulerImpl: Adding task set 655.0 with 8 tasks
16/09/07 19:10:30 INFO TaskSetManager: Starting task 0.0 in stage 655.0 (TID 5240, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 1.0 in stage 655.0 (TID 5241, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 2.0 in stage 655.0 (TID 5242, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 3.0 in stage 655.0 (TID 5243, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 4.0 in stage 655.0 (TID 5244, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 5.0 in stage 655.0 (TID 5245, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 6.0 in stage 655.0 (TID 5246, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:30 INFO TaskSetManager: Starting task 7.0 in stage 655.0 (TID 5247, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:30 INFO Executor: Running task 0.0 in stage 655.0 (TID 5240)
16/09/07 19:10:30 INFO Executor: Running task 3.0 in stage 655.0 (TID 5243)
16/09/07 19:10:30 INFO Executor: Running task 7.0 in stage 655.0 (TID 5247)
16/09/07 19:10:30 INFO Executor: Running task 6.0 in stage 655.0 (TID 5246)
16/09/07 19:10:30 INFO Executor: Running task 1.0 in stage 655.0 (TID 5241)
16/09/07 19:10:30 INFO Executor: Running task 4.0 in stage 655.0 (TID 5244)
16/09/07 19:10:30 INFO Executor: Running task 5.0 in stage 655.0 (TID 5245)
16/09/07 19:10:30 INFO Executor: Running task 2.0 in stage 655.0 (TID 5242)
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:30 INFO Executor: Finished task 5.0 in stage 655.0 (TID 5245). 2160 bytes result sent to driver
16/09/07 19:10:30 INFO Executor: Finished task 2.0 in stage 655.0 (TID 5242). 2160 bytes result sent to driver
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:30 INFO TaskSetManager: Finished task 2.0 in stage 655.0 (TID 5242) in 2 ms on localhost (1/8)
16/09/07 19:10:30 INFO TaskSetManager: Finished task 5.0 in stage 655.0 (TID 5245) in 2 ms on localhost (2/8)
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:30 INFO Executor: Finished task 4.0 in stage 655.0 (TID 5244). 2160 bytes result sent to driver
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:30 INFO TaskSetManager: Finished task 4.0 in stage 655.0 (TID 5244) in 3 ms on localhost (3/8)
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:30 INFO Executor: Finished task 3.0 in stage 655.0 (TID 5243). 2160 bytes result sent to driver
16/09/07 19:10:30 INFO TaskSetManager: Finished task 3.0 in stage 655.0 (TID 5243) in 5 ms on localhost (4/8)
16/09/07 19:10:30 INFO BlockManagerInfo: Removed broadcast_990_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:30 INFO Executor: Finished task 1.0 in stage 655.0 (TID 5241). 3485274 bytes result sent to driver
16/09/07 19:10:30 INFO TaskSetManager: Finished task 1.0 in stage 655.0 (TID 5241) in 598 ms on localhost (5/8)
16/09/07 19:10:31 INFO Executor: Finished task 0.0 in stage 655.0 (TID 5240). 3485274 bytes result sent to driver
16/09/07 19:10:31 INFO Executor: Finished task 6.0 in stage 655.0 (TID 5246). 3485274 bytes result sent to driver
16/09/07 19:10:31 INFO Executor: Finished task 7.0 in stage 655.0 (TID 5247). 3485274 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 0.0 in stage 655.0 (TID 5240) in 912 ms on localhost (6/8)
16/09/07 19:10:31 INFO TaskSetManager: Finished task 6.0 in stage 655.0 (TID 5246) in 913 ms on localhost (7/8)
16/09/07 19:10:31 INFO TaskSetManager: Finished task 7.0 in stage 655.0 (TID 5247) in 917 ms on localhost (8/8)
16/09/07 19:10:31 INFO DAGScheduler: ResultStage 655 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.917 s
16/09/07 19:10:31 INFO TaskSchedulerImpl: Removed TaskSet 655.0, whose tasks have all completed, from pool 
16/09/07 19:10:31 INFO DAGScheduler: Job 332 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.925319 s
16/09/07 19:10:31 INFO ParameterAveragingTrainingMaster: Completed training of split 11 of 78
16/09/07 19:10:31 INFO ParameterAveragingTrainingMaster: Starting training of split 12 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:31 INFO MemoryStore: ensureFreeSpace(18027848) called with curMem=18841923, maxMem=4042889625
16/09/07 19:10:31 INFO MemoryStore: Block broadcast_992 stored as values in memory (estimated size 17.2 MB, free 3.7 GB)
16/09/07 19:10:31 INFO BlockManagerInfo: Removed broadcast_991_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:31 INFO ContextCleaner: Cleaned accumulator 656
16/09/07 19:10:31 INFO ContextCleaner: Cleaned accumulator 655
16/09/07 19:10:31 INFO BlockManagerInfo: Removed broadcast_989_piece0 on localhost:41251 in memory (size: 269.9 KB, free: 3.8 GB)
16/09/07 19:10:31 INFO ContextCleaner: Cleaned shuffle 322
16/09/07 19:10:31 INFO MemoryStore: ensureFreeSpace(277318) called with curMem=18571672, maxMem=4042889625
16/09/07 19:10:31 INFO MemoryStore: Block broadcast_992_piece0 stored as bytes in memory (estimated size 270.8 KB, free 3.7 GB)
16/09/07 19:10:31 INFO BlockManagerInfo: Added broadcast_992_piece0 in memory on localhost:41251 (size: 270.8 KB, free: 3.8 GB)
16/09/07 19:10:31 INFO SparkContext: Created broadcast 992 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:31 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:31 INFO DAGScheduler: Registering RDD 2023 (repartition at SparkUtils.java:190)
16/09/07 19:10:31 INFO DAGScheduler: Got job 333 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:31 INFO DAGScheduler: Final stage: ResultStage 657(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:31 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 656)
16/09/07 19:10:31 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 656)
16/09/07 19:10:31 INFO DAGScheduler: Submitting ShuffleMapStage 656 (MapPartitionsRDD[2023] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:31 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18848990, maxMem=4042889625
16/09/07 19:10:31 INFO MemoryStore: Block broadcast_993 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:31 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18853454, maxMem=4042889625
16/09/07 19:10:31 INFO MemoryStore: Block broadcast_993_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:31 INFO BlockManagerInfo: Added broadcast_993_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:31 INFO SparkContext: Created broadcast 993 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:31 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 656 (MapPartitionsRDD[2023] at repartition at SparkUtils.java:190)
16/09/07 19:10:31 INFO TaskSchedulerImpl: Adding task set 656.0 with 8 tasks
16/09/07 19:10:31 INFO TaskSetManager: Starting task 0.0 in stage 656.0 (TID 5248, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 1.0 in stage 656.0 (TID 5249, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 2.0 in stage 656.0 (TID 5250, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 3.0 in stage 656.0 (TID 5251, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 4.0 in stage 656.0 (TID 5252, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 5.0 in stage 656.0 (TID 5253, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 6.0 in stage 656.0 (TID 5254, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 7.0 in stage 656.0 (TID 5255, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:31 INFO Executor: Running task 3.0 in stage 656.0 (TID 5251)
16/09/07 19:10:31 INFO Executor: Running task 1.0 in stage 656.0 (TID 5249)
16/09/07 19:10:31 INFO Executor: Running task 4.0 in stage 656.0 (TID 5252)
16/09/07 19:10:31 INFO Executor: Running task 6.0 in stage 656.0 (TID 5254)
16/09/07 19:10:31 INFO Executor: Running task 5.0 in stage 656.0 (TID 5253)
16/09/07 19:10:31 INFO Executor: Running task 0.0 in stage 656.0 (TID 5248)
16/09/07 19:10:31 INFO Executor: Running task 7.0 in stage 656.0 (TID 5255)
16/09/07 19:10:31 INFO Executor: Running task 2.0 in stage 656.0 (TID 5250)
16/09/07 19:10:31 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:31 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:31 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:31 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:31 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:31 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:31 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:31 INFO Executor: Finished task 3.0 in stage 656.0 (TID 5251). 2260 bytes result sent to driver
16/09/07 19:10:31 INFO Executor: Finished task 6.0 in stage 656.0 (TID 5254). 2260 bytes result sent to driver
16/09/07 19:10:31 INFO Executor: Finished task 0.0 in stage 656.0 (TID 5248). 2260 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 6.0 in stage 656.0 (TID 5254) in 2 ms on localhost (1/8)
16/09/07 19:10:31 INFO Executor: Finished task 4.0 in stage 656.0 (TID 5252). 2260 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 3.0 in stage 656.0 (TID 5251) in 3 ms on localhost (2/8)
16/09/07 19:10:31 INFO TaskSetManager: Finished task 0.0 in stage 656.0 (TID 5248) in 3 ms on localhost (3/8)
16/09/07 19:10:31 INFO Executor: Finished task 5.0 in stage 656.0 (TID 5253). 2260 bytes result sent to driver
16/09/07 19:10:31 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:31 INFO Executor: Finished task 2.0 in stage 656.0 (TID 5250). 2260 bytes result sent to driver
16/09/07 19:10:31 INFO Executor: Finished task 7.0 in stage 656.0 (TID 5255). 2260 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 4.0 in stage 656.0 (TID 5252) in 3 ms on localhost (4/8)
16/09/07 19:10:31 INFO TaskSetManager: Finished task 5.0 in stage 656.0 (TID 5253) in 3 ms on localhost (5/8)
16/09/07 19:10:31 INFO TaskSetManager: Finished task 2.0 in stage 656.0 (TID 5250) in 3 ms on localhost (6/8)
16/09/07 19:10:31 INFO TaskSetManager: Finished task 7.0 in stage 656.0 (TID 5255) in 3 ms on localhost (7/8)
16/09/07 19:10:31 INFO Executor: Finished task 1.0 in stage 656.0 (TID 5249). 2260 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 1.0 in stage 656.0 (TID 5249) in 4 ms on localhost (8/8)
16/09/07 19:10:31 INFO TaskSchedulerImpl: Removed TaskSet 656.0, whose tasks have all completed, from pool 
16/09/07 19:10:31 INFO DAGScheduler: ShuffleMapStage 656 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:31 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:31 INFO DAGScheduler: running: Set()
16/09/07 19:10:31 INFO DAGScheduler: waiting: Set(ResultStage 657)
16/09/07 19:10:31 INFO DAGScheduler: failed: Set()
16/09/07 19:10:31 INFO DAGScheduler: Missing parents for ResultStage 657: List()
16/09/07 19:10:31 INFO DAGScheduler: Submitting ResultStage 657 (MapPartitionsRDD[2027] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:31 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18856018, maxMem=4042889625
16/09/07 19:10:31 INFO MemoryStore: Block broadcast_994 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:31 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18861114, maxMem=4042889625
16/09/07 19:10:31 INFO MemoryStore: Block broadcast_994_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:31 INFO BlockManagerInfo: Added broadcast_994_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:31 INFO SparkContext: Created broadcast 994 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:31 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 657 (MapPartitionsRDD[2027] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:31 INFO TaskSchedulerImpl: Adding task set 657.0 with 8 tasks
16/09/07 19:10:31 INFO TaskSetManager: Starting task 0.0 in stage 657.0 (TID 5256, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 1.0 in stage 657.0 (TID 5257, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 2.0 in stage 657.0 (TID 5258, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 3.0 in stage 657.0 (TID 5259, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 4.0 in stage 657.0 (TID 5260, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 5.0 in stage 657.0 (TID 5261, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 6.0 in stage 657.0 (TID 5262, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:31 INFO TaskSetManager: Starting task 7.0 in stage 657.0 (TID 5263, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:31 INFO Executor: Running task 5.0 in stage 657.0 (TID 5261)
16/09/07 19:10:31 INFO Executor: Running task 6.0 in stage 657.0 (TID 5262)
16/09/07 19:10:31 INFO Executor: Running task 7.0 in stage 657.0 (TID 5263)
16/09/07 19:10:31 INFO Executor: Running task 4.0 in stage 657.0 (TID 5260)
16/09/07 19:10:31 INFO Executor: Running task 0.0 in stage 657.0 (TID 5256)
16/09/07 19:10:31 INFO Executor: Running task 3.0 in stage 657.0 (TID 5259)
16/09/07 19:10:31 INFO Executor: Running task 1.0 in stage 657.0 (TID 5257)
16/09/07 19:10:31 INFO Executor: Running task 2.0 in stage 657.0 (TID 5258)
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:31 INFO Executor: Finished task 5.0 in stage 657.0 (TID 5261). 2160 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 5.0 in stage 657.0 (TID 5261) in 2 ms on localhost (1/8)
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:31 INFO Executor: Finished task 3.0 in stage 657.0 (TID 5259). 2160 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 3.0 in stage 657.0 (TID 5259) in 3 ms on localhost (2/8)
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:31 INFO Executor: Finished task 4.0 in stage 657.0 (TID 5260). 2160 bytes result sent to driver
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:31 INFO TaskSetManager: Finished task 4.0 in stage 657.0 (TID 5260) in 5 ms on localhost (3/8)
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:31 INFO Executor: Finished task 2.0 in stage 657.0 (TID 5258). 2160 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 2.0 in stage 657.0 (TID 5258) in 7 ms on localhost (4/8)
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:31 INFO Executor: Finished task 1.0 in stage 657.0 (TID 5257). 2160 bytes result sent to driver
16/09/07 19:10:31 INFO TaskSetManager: Finished task 1.0 in stage 657.0 (TID 5257) in 8 ms on localhost (5/8)
16/09/07 19:10:31 INFO BlockManagerInfo: Removed broadcast_993_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:32 INFO Executor: Finished task 0.0 in stage 657.0 (TID 5256). 3485274 bytes result sent to driver
16/09/07 19:10:32 INFO TaskSetManager: Finished task 0.0 in stage 657.0 (TID 5256) in 871 ms on localhost (6/8)
16/09/07 19:10:32 INFO Executor: Finished task 6.0 in stage 657.0 (TID 5262). 3485524 bytes result sent to driver
16/09/07 19:10:32 INFO Executor: Finished task 7.0 in stage 657.0 (TID 5263). 3485524 bytes result sent to driver
16/09/07 19:10:32 INFO TaskSetManager: Finished task 6.0 in stage 657.0 (TID 5262) in 943 ms on localhost (7/8)
16/09/07 19:10:32 INFO TaskSetManager: Finished task 7.0 in stage 657.0 (TID 5263) in 943 ms on localhost (8/8)
16/09/07 19:10:32 INFO TaskSchedulerImpl: Removed TaskSet 657.0, whose tasks have all completed, from pool 
16/09/07 19:10:32 INFO DAGScheduler: ResultStage 657 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.944 s
16/09/07 19:10:32 INFO DAGScheduler: Job 333 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.951000 s
16/09/07 19:10:32 INFO ParameterAveragingTrainingMaster: Completed training of split 12 of 78
16/09/07 19:10:32 INFO ParameterAveragingTrainingMaster: Starting training of split 13 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:32 INFO BlockManagerInfo: Removed broadcast_994_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:32 INFO ContextCleaner: Cleaned accumulator 658
16/09/07 19:10:32 INFO ContextCleaner: Cleaned accumulator 657
16/09/07 19:10:32 INFO BlockManagerInfo: Removed broadcast_992_piece0 on localhost:41251 in memory (size: 270.8 KB, free: 3.8 GB)
16/09/07 19:10:32 INFO ContextCleaner: Cleaned shuffle 323
16/09/07 19:10:32 INFO MemoryStore: ensureFreeSpace(17746256) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:32 INFO MemoryStore: Block broadcast_995 stored as values in memory (estimated size 16.9 MB, free 3.7 GB)
16/09/07 19:10:32 INFO MemoryStore: ensureFreeSpace(276780) called with curMem=18290080, maxMem=4042889625
16/09/07 19:10:32 INFO MemoryStore: Block broadcast_995_piece0 stored as bytes in memory (estimated size 270.3 KB, free 3.7 GB)
16/09/07 19:10:32 INFO BlockManagerInfo: Added broadcast_995_piece0 in memory on localhost:41251 (size: 270.3 KB, free: 3.8 GB)
16/09/07 19:10:32 INFO SparkContext: Created broadcast 995 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:32 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:32 INFO DAGScheduler: Registering RDD 2028 (repartition at SparkUtils.java:190)
16/09/07 19:10:32 INFO DAGScheduler: Got job 334 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:32 INFO DAGScheduler: Final stage: ResultStage 659(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:32 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 658)
16/09/07 19:10:32 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 658)
16/09/07 19:10:32 INFO DAGScheduler: Submitting ShuffleMapStage 658 (MapPartitionsRDD[2028] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:32 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18566860, maxMem=4042889625
16/09/07 19:10:32 INFO MemoryStore: Block broadcast_996 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:32 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18571324, maxMem=4042889625
16/09/07 19:10:32 INFO MemoryStore: Block broadcast_996_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:32 INFO BlockManagerInfo: Added broadcast_996_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:32 INFO SparkContext: Created broadcast 996 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:32 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 658 (MapPartitionsRDD[2028] at repartition at SparkUtils.java:190)
16/09/07 19:10:32 INFO TaskSchedulerImpl: Adding task set 658.0 with 8 tasks
16/09/07 19:10:32 INFO TaskSetManager: Starting task 0.0 in stage 658.0 (TID 5264, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 1.0 in stage 658.0 (TID 5265, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 2.0 in stage 658.0 (TID 5266, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 3.0 in stage 658.0 (TID 5267, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 4.0 in stage 658.0 (TID 5268, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 5.0 in stage 658.0 (TID 5269, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 6.0 in stage 658.0 (TID 5270, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 7.0 in stage 658.0 (TID 5271, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:32 INFO Executor: Running task 1.0 in stage 658.0 (TID 5265)
16/09/07 19:10:32 INFO Executor: Running task 2.0 in stage 658.0 (TID 5266)
16/09/07 19:10:32 INFO Executor: Running task 6.0 in stage 658.0 (TID 5270)
16/09/07 19:10:32 INFO Executor: Running task 5.0 in stage 658.0 (TID 5269)
16/09/07 19:10:32 INFO Executor: Running task 0.0 in stage 658.0 (TID 5264)
16/09/07 19:10:32 INFO Executor: Running task 7.0 in stage 658.0 (TID 5271)
16/09/07 19:10:32 INFO Executor: Running task 4.0 in stage 658.0 (TID 5268)
16/09/07 19:10:32 INFO Executor: Running task 3.0 in stage 658.0 (TID 5267)
16/09/07 19:10:32 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:32 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:32 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:32 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:32 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:32 INFO Executor: Finished task 2.0 in stage 658.0 (TID 5266). 2260 bytes result sent to driver
16/09/07 19:10:32 INFO Executor: Finished task 0.0 in stage 658.0 (TID 5264). 2260 bytes result sent to driver
16/09/07 19:10:32 INFO Executor: Finished task 4.0 in stage 658.0 (TID 5268). 2260 bytes result sent to driver
16/09/07 19:10:32 INFO TaskSetManager: Finished task 0.0 in stage 658.0 (TID 5264) in 3 ms on localhost (1/8)
16/09/07 19:10:32 INFO TaskSetManager: Finished task 2.0 in stage 658.0 (TID 5266) in 2 ms on localhost (2/8)
16/09/07 19:10:32 INFO TaskSetManager: Finished task 4.0 in stage 658.0 (TID 5268) in 2 ms on localhost (3/8)
16/09/07 19:10:32 INFO Executor: Finished task 5.0 in stage 658.0 (TID 5269). 2260 bytes result sent to driver
16/09/07 19:10:32 INFO TaskSetManager: Finished task 5.0 in stage 658.0 (TID 5269) in 3 ms on localhost (4/8)
16/09/07 19:10:32 INFO Executor: Finished task 6.0 in stage 658.0 (TID 5270). 2260 bytes result sent to driver
16/09/07 19:10:32 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:32 INFO TaskSetManager: Finished task 6.0 in stage 658.0 (TID 5270) in 3 ms on localhost (5/8)
16/09/07 19:10:32 INFO Executor: Finished task 1.0 in stage 658.0 (TID 5265). 2260 bytes result sent to driver
16/09/07 19:10:32 INFO TaskSetManager: Finished task 1.0 in stage 658.0 (TID 5265) in 4 ms on localhost (6/8)
16/09/07 19:10:32 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:32 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:32 INFO Executor: Finished task 3.0 in stage 658.0 (TID 5267). 2260 bytes result sent to driver
16/09/07 19:10:32 INFO Executor: Finished task 7.0 in stage 658.0 (TID 5271). 2260 bytes result sent to driver
16/09/07 19:10:32 INFO TaskSetManager: Finished task 3.0 in stage 658.0 (TID 5267) in 5 ms on localhost (7/8)
16/09/07 19:10:32 INFO TaskSetManager: Finished task 7.0 in stage 658.0 (TID 5271) in 5 ms on localhost (8/8)
16/09/07 19:10:32 INFO TaskSchedulerImpl: Removed TaskSet 658.0, whose tasks have all completed, from pool 
16/09/07 19:10:32 INFO DAGScheduler: ShuffleMapStage 658 (repartition at SparkUtils.java:190) finished in 0.006 s
16/09/07 19:10:32 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:32 INFO DAGScheduler: running: Set()
16/09/07 19:10:32 INFO DAGScheduler: waiting: Set(ResultStage 659)
16/09/07 19:10:32 INFO DAGScheduler: failed: Set()
16/09/07 19:10:32 INFO DAGScheduler: Missing parents for ResultStage 659: List()
16/09/07 19:10:32 INFO DAGScheduler: Submitting ResultStage 659 (MapPartitionsRDD[2032] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:32 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18573888, maxMem=4042889625
16/09/07 19:10:32 INFO MemoryStore: Block broadcast_997 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:32 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18578984, maxMem=4042889625
16/09/07 19:10:32 INFO MemoryStore: Block broadcast_997_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:32 INFO BlockManagerInfo: Added broadcast_997_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:32 INFO SparkContext: Created broadcast 997 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:32 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 659 (MapPartitionsRDD[2032] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:32 INFO TaskSchedulerImpl: Adding task set 659.0 with 8 tasks
16/09/07 19:10:32 INFO TaskSetManager: Starting task 0.0 in stage 659.0 (TID 5272, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 1.0 in stage 659.0 (TID 5273, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 2.0 in stage 659.0 (TID 5274, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 3.0 in stage 659.0 (TID 5275, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 4.0 in stage 659.0 (TID 5276, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 5.0 in stage 659.0 (TID 5277, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 6.0 in stage 659.0 (TID 5278, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:32 INFO TaskSetManager: Starting task 7.0 in stage 659.0 (TID 5279, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:32 INFO Executor: Running task 0.0 in stage 659.0 (TID 5272)
16/09/07 19:10:32 INFO Executor: Running task 3.0 in stage 659.0 (TID 5275)
16/09/07 19:10:32 INFO Executor: Running task 7.0 in stage 659.0 (TID 5279)
16/09/07 19:10:32 INFO Executor: Running task 6.0 in stage 659.0 (TID 5278)
16/09/07 19:10:32 INFO Executor: Running task 4.0 in stage 659.0 (TID 5276)
16/09/07 19:10:32 INFO Executor: Running task 1.0 in stage 659.0 (TID 5273)
16/09/07 19:10:32 INFO Executor: Running task 5.0 in stage 659.0 (TID 5277)
16/09/07 19:10:32 INFO Executor: Running task 2.0 in stage 659.0 (TID 5274)
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:32 INFO Executor: Finished task 5.0 in stage 659.0 (TID 5277). 2160 bytes result sent to driver
16/09/07 19:10:32 INFO Executor: Finished task 1.0 in stage 659.0 (TID 5273). 2160 bytes result sent to driver
16/09/07 19:10:32 INFO Executor: Finished task 2.0 in stage 659.0 (TID 5274). 2160 bytes result sent to driver
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:32 INFO TaskSetManager: Finished task 2.0 in stage 659.0 (TID 5274) in 2 ms on localhost (1/8)
16/09/07 19:10:32 INFO TaskSetManager: Finished task 5.0 in stage 659.0 (TID 5277) in 2 ms on localhost (2/8)
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:32 INFO TaskSetManager: Finished task 1.0 in stage 659.0 (TID 5273) in 2 ms on localhost (3/8)
16/09/07 19:10:32 INFO Executor: Finished task 4.0 in stage 659.0 (TID 5276). 2160 bytes result sent to driver
16/09/07 19:10:32 INFO TaskSetManager: Finished task 4.0 in stage 659.0 (TID 5276) in 2 ms on localhost (4/8)
16/09/07 19:10:32 INFO Executor: Finished task 3.0 in stage 659.0 (TID 5275). 2160 bytes result sent to driver
16/09/07 19:10:32 INFO TaskSetManager: Finished task 3.0 in stage 659.0 (TID 5275) in 2 ms on localhost (5/8)
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:32 INFO BlockManagerInfo: Removed broadcast_996_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:33 INFO Executor: Finished task 6.0 in stage 659.0 (TID 5278). 3485274 bytes result sent to driver
16/09/07 19:10:33 INFO Executor: Finished task 7.0 in stage 659.0 (TID 5279). 3485274 bytes result sent to driver
16/09/07 19:10:33 INFO Executor: Finished task 0.0 in stage 659.0 (TID 5272). 3485274 bytes result sent to driver
16/09/07 19:10:33 INFO TaskSetManager: Finished task 6.0 in stage 659.0 (TID 5278) in 895 ms on localhost (6/8)
16/09/07 19:10:33 INFO TaskSetManager: Finished task 7.0 in stage 659.0 (TID 5279) in 908 ms on localhost (7/8)
16/09/07 19:10:33 INFO TaskSetManager: Finished task 0.0 in stage 659.0 (TID 5272) in 911 ms on localhost (8/8)
16/09/07 19:10:33 INFO TaskSchedulerImpl: Removed TaskSet 659.0, whose tasks have all completed, from pool 
16/09/07 19:10:33 INFO DAGScheduler: ResultStage 659 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.911 s
16/09/07 19:10:33 INFO DAGScheduler: Job 334 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.920919 s
16/09/07 19:10:33 INFO ParameterAveragingTrainingMaster: Completed training of split 13 of 78
16/09/07 19:10:33 INFO ParameterAveragingTrainingMaster: Starting training of split 14 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:33 INFO MemoryStore: ensureFreeSpace(18253592) called with curMem=18574699, maxMem=4042889625
16/09/07 19:10:33 INFO MemoryStore: Block broadcast_998 stored as values in memory (estimated size 17.4 MB, free 3.7 GB)
16/09/07 19:10:33 INFO BlockManagerInfo: Removed broadcast_997_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:33 INFO ContextCleaner: Cleaned accumulator 660
16/09/07 19:10:33 INFO ContextCleaner: Cleaned accumulator 659
16/09/07 19:10:33 INFO BlockManagerInfo: Removed broadcast_995_piece0 on localhost:41251 in memory (size: 270.3 KB, free: 3.8 GB)
16/09/07 19:10:33 INFO ContextCleaner: Cleaned shuffle 324
16/09/07 19:10:33 INFO MemoryStore: ensureFreeSpace(276778) called with curMem=18797416, maxMem=4042889625
16/09/07 19:10:33 INFO MemoryStore: Block broadcast_998_piece0 stored as bytes in memory (estimated size 270.3 KB, free 3.7 GB)
16/09/07 19:10:33 INFO BlockManagerInfo: Added broadcast_998_piece0 in memory on localhost:41251 (size: 270.3 KB, free: 3.8 GB)
16/09/07 19:10:33 INFO SparkContext: Created broadcast 998 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:33 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:33 INFO DAGScheduler: Registering RDD 2033 (repartition at SparkUtils.java:190)
16/09/07 19:10:33 INFO DAGScheduler: Got job 335 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:33 INFO DAGScheduler: Final stage: ResultStage 661(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:33 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 660)
16/09/07 19:10:33 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 660)
16/09/07 19:10:33 INFO DAGScheduler: Submitting ShuffleMapStage 660 (MapPartitionsRDD[2033] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:33 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19074194, maxMem=4042889625
16/09/07 19:10:33 INFO MemoryStore: Block broadcast_999 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:33 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=19078658, maxMem=4042889625
16/09/07 19:10:33 INFO MemoryStore: Block broadcast_999_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:33 INFO BlockManagerInfo: Added broadcast_999_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:33 INFO SparkContext: Created broadcast 999 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:33 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 660 (MapPartitionsRDD[2033] at repartition at SparkUtils.java:190)
16/09/07 19:10:33 INFO TaskSchedulerImpl: Adding task set 660.0 with 8 tasks
16/09/07 19:10:33 INFO TaskSetManager: Starting task 0.0 in stage 660.0 (TID 5280, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 1.0 in stage 660.0 (TID 5281, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 2.0 in stage 660.0 (TID 5282, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 3.0 in stage 660.0 (TID 5283, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 4.0 in stage 660.0 (TID 5284, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 5.0 in stage 660.0 (TID 5285, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 6.0 in stage 660.0 (TID 5286, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 7.0 in stage 660.0 (TID 5287, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:33 INFO Executor: Running task 2.0 in stage 660.0 (TID 5282)
16/09/07 19:10:33 INFO Executor: Running task 0.0 in stage 660.0 (TID 5280)
16/09/07 19:10:33 INFO Executor: Running task 5.0 in stage 660.0 (TID 5285)
16/09/07 19:10:33 INFO Executor: Running task 3.0 in stage 660.0 (TID 5283)
16/09/07 19:10:33 INFO Executor: Running task 4.0 in stage 660.0 (TID 5284)
16/09/07 19:10:33 INFO Executor: Running task 7.0 in stage 660.0 (TID 5287)
16/09/07 19:10:33 INFO Executor: Running task 1.0 in stage 660.0 (TID 5281)
16/09/07 19:10:33 INFO Executor: Running task 6.0 in stage 660.0 (TID 5286)
16/09/07 19:10:33 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:33 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:33 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:33 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:33 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:33 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:33 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:33 INFO Executor: Finished task 1.0 in stage 660.0 (TID 5281). 2260 bytes result sent to driver
16/09/07 19:10:33 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:33 INFO TaskSetManager: Finished task 1.0 in stage 660.0 (TID 5281) in 2 ms on localhost (1/8)
16/09/07 19:10:33 INFO Executor: Finished task 0.0 in stage 660.0 (TID 5280). 2260 bytes result sent to driver
16/09/07 19:10:33 INFO Executor: Finished task 4.0 in stage 660.0 (TID 5284). 2260 bytes result sent to driver
16/09/07 19:10:33 INFO TaskSetManager: Finished task 0.0 in stage 660.0 (TID 5280) in 3 ms on localhost (2/8)
16/09/07 19:10:33 INFO Executor: Finished task 7.0 in stage 660.0 (TID 5287). 2260 bytes result sent to driver
16/09/07 19:10:33 INFO Executor: Finished task 2.0 in stage 660.0 (TID 5282). 2260 bytes result sent to driver
16/09/07 19:10:33 INFO Executor: Finished task 6.0 in stage 660.0 (TID 5286). 2260 bytes result sent to driver
16/09/07 19:10:33 INFO Executor: Finished task 3.0 in stage 660.0 (TID 5283). 2260 bytes result sent to driver
16/09/07 19:10:33 INFO TaskSetManager: Finished task 4.0 in stage 660.0 (TID 5284) in 3 ms on localhost (3/8)
16/09/07 19:10:33 INFO TaskSetManager: Finished task 7.0 in stage 660.0 (TID 5287) in 3 ms on localhost (4/8)
16/09/07 19:10:33 INFO TaskSetManager: Finished task 2.0 in stage 660.0 (TID 5282) in 3 ms on localhost (5/8)
16/09/07 19:10:33 INFO TaskSetManager: Finished task 6.0 in stage 660.0 (TID 5286) in 3 ms on localhost (6/8)
16/09/07 19:10:33 INFO TaskSetManager: Finished task 3.0 in stage 660.0 (TID 5283) in 3 ms on localhost (7/8)
16/09/07 19:10:33 INFO Executor: Finished task 5.0 in stage 660.0 (TID 5285). 2260 bytes result sent to driver
16/09/07 19:10:33 INFO TaskSetManager: Finished task 5.0 in stage 660.0 (TID 5285) in 3 ms on localhost (8/8)
16/09/07 19:10:33 INFO TaskSchedulerImpl: Removed TaskSet 660.0, whose tasks have all completed, from pool 
16/09/07 19:10:33 INFO DAGScheduler: ShuffleMapStage 660 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:33 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:33 INFO DAGScheduler: running: Set()
16/09/07 19:10:33 INFO DAGScheduler: waiting: Set(ResultStage 661)
16/09/07 19:10:33 INFO DAGScheduler: failed: Set()
16/09/07 19:10:33 INFO DAGScheduler: Missing parents for ResultStage 661: List()
16/09/07 19:10:33 INFO DAGScheduler: Submitting ResultStage 661 (MapPartitionsRDD[2037] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:33 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19081220, maxMem=4042889625
16/09/07 19:10:33 INFO MemoryStore: Block broadcast_1000 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:33 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19086316, maxMem=4042889625
16/09/07 19:10:33 INFO MemoryStore: Block broadcast_1000_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:33 INFO BlockManagerInfo: Added broadcast_1000_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:33 INFO SparkContext: Created broadcast 1000 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:33 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 661 (MapPartitionsRDD[2037] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:33 INFO TaskSchedulerImpl: Adding task set 661.0 with 8 tasks
16/09/07 19:10:33 INFO TaskSetManager: Starting task 0.0 in stage 661.0 (TID 5288, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 1.0 in stage 661.0 (TID 5289, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 2.0 in stage 661.0 (TID 5290, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 3.0 in stage 661.0 (TID 5291, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 4.0 in stage 661.0 (TID 5292, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 5.0 in stage 661.0 (TID 5293, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 6.0 in stage 661.0 (TID 5294, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:33 INFO TaskSetManager: Starting task 7.0 in stage 661.0 (TID 5295, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:33 INFO Executor: Running task 2.0 in stage 661.0 (TID 5290)
16/09/07 19:10:33 INFO Executor: Running task 0.0 in stage 661.0 (TID 5288)
16/09/07 19:10:33 INFO Executor: Running task 6.0 in stage 661.0 (TID 5294)
16/09/07 19:10:33 INFO Executor: Running task 1.0 in stage 661.0 (TID 5289)
16/09/07 19:10:33 INFO Executor: Running task 3.0 in stage 661.0 (TID 5291)
16/09/07 19:10:33 INFO Executor: Running task 7.0 in stage 661.0 (TID 5295)
16/09/07 19:10:33 INFO Executor: Running task 4.0 in stage 661.0 (TID 5292)
16/09/07 19:10:33 INFO Executor: Running task 5.0 in stage 661.0 (TID 5293)
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:33 INFO Executor: Finished task 1.0 in stage 661.0 (TID 5289). 2160 bytes result sent to driver
16/09/07 19:10:33 INFO TaskSetManager: Finished task 1.0 in stage 661.0 (TID 5289) in 1 ms on localhost (1/8)
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:33 INFO Executor: Finished task 5.0 in stage 661.0 (TID 5293). 2160 bytes result sent to driver
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:33 INFO TaskSetManager: Finished task 5.0 in stage 661.0 (TID 5293) in 2 ms on localhost (2/8)
16/09/07 19:10:33 INFO Executor: Finished task 4.0 in stage 661.0 (TID 5292). 2160 bytes result sent to driver
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:33 INFO TaskSetManager: Finished task 4.0 in stage 661.0 (TID 5292) in 2 ms on localhost (3/8)
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:33 INFO Executor: Finished task 3.0 in stage 661.0 (TID 5291). 2160 bytes result sent to driver
16/09/07 19:10:33 INFO TaskSetManager: Finished task 3.0 in stage 661.0 (TID 5291) in 2 ms on localhost (4/8)
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:33 INFO Executor: Finished task 2.0 in stage 661.0 (TID 5290). 2160 bytes result sent to driver
16/09/07 19:10:33 INFO TaskSetManager: Finished task 2.0 in stage 661.0 (TID 5290) in 2 ms on localhost (5/8)
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:33 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:33 INFO BlockManagerInfo: Removed broadcast_999_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:34 INFO Executor: Finished task 0.0 in stage 661.0 (TID 5288). 3485274 bytes result sent to driver
16/09/07 19:10:34 INFO TaskSetManager: Finished task 0.0 in stage 661.0 (TID 5288) in 822 ms on localhost (6/8)
16/09/07 19:10:34 INFO Executor: Finished task 7.0 in stage 661.0 (TID 5295). 3485274 bytes result sent to driver
16/09/07 19:10:34 INFO Executor: Finished task 6.0 in stage 661.0 (TID 5294). 3485274 bytes result sent to driver
16/09/07 19:10:34 INFO TaskSetManager: Finished task 7.0 in stage 661.0 (TID 5295) in 1003 ms on localhost (7/8)
16/09/07 19:10:34 INFO TaskSetManager: Finished task 6.0 in stage 661.0 (TID 5294) in 1005 ms on localhost (8/8)
16/09/07 19:10:34 INFO TaskSchedulerImpl: Removed TaskSet 661.0, whose tasks have all completed, from pool 
16/09/07 19:10:34 INFO DAGScheduler: ResultStage 661 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 1.005 s
16/09/07 19:10:34 INFO DAGScheduler: Job 335 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.012949 s
16/09/07 19:10:34 INFO ParameterAveragingTrainingMaster: Completed training of split 14 of 78
16/09/07 19:10:34 INFO ParameterAveragingTrainingMaster: Starting training of split 15 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:34 INFO MemoryStore: ensureFreeSpace(18108696) called with curMem=19082033, maxMem=4042889625
16/09/07 19:10:34 INFO MemoryStore: Block broadcast_1001 stored as values in memory (estimated size 17.3 MB, free 3.7 GB)
16/09/07 19:10:34 INFO BlockManagerInfo: Removed broadcast_1000_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:34 INFO ContextCleaner: Cleaned accumulator 662
16/09/07 19:10:34 INFO ContextCleaner: Cleaned accumulator 661
16/09/07 19:10:34 INFO BlockManagerInfo: Removed broadcast_998_piece0 on localhost:41251 in memory (size: 270.3 KB, free: 3.8 GB)
16/09/07 19:10:34 INFO ContextCleaner: Cleaned shuffle 325
16/09/07 19:10:35 INFO MemoryStore: ensureFreeSpace(276123) called with curMem=18652520, maxMem=4042889625
16/09/07 19:10:35 INFO MemoryStore: Block broadcast_1001_piece0 stored as bytes in memory (estimated size 269.7 KB, free 3.7 GB)
16/09/07 19:10:35 INFO BlockManagerInfo: Added broadcast_1001_piece0 in memory on localhost:41251 (size: 269.7 KB, free: 3.8 GB)
16/09/07 19:10:35 INFO SparkContext: Created broadcast 1001 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:35 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:35 INFO DAGScheduler: Registering RDD 2038 (repartition at SparkUtils.java:190)
16/09/07 19:10:35 INFO DAGScheduler: Got job 336 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:35 INFO DAGScheduler: Final stage: ResultStage 663(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:35 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 662)
16/09/07 19:10:35 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 662)
16/09/07 19:10:35 INFO DAGScheduler: Submitting ShuffleMapStage 662 (MapPartitionsRDD[2038] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:35 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18928643, maxMem=4042889625
16/09/07 19:10:35 INFO MemoryStore: Block broadcast_1002 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:35 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18933107, maxMem=4042889625
16/09/07 19:10:35 INFO MemoryStore: Block broadcast_1002_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:35 INFO BlockManagerInfo: Added broadcast_1002_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:35 INFO SparkContext: Created broadcast 1002 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:35 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 662 (MapPartitionsRDD[2038] at repartition at SparkUtils.java:190)
16/09/07 19:10:35 INFO TaskSchedulerImpl: Adding task set 662.0 with 8 tasks
16/09/07 19:10:35 INFO TaskSetManager: Starting task 0.0 in stage 662.0 (TID 5296, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 1.0 in stage 662.0 (TID 5297, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 2.0 in stage 662.0 (TID 5298, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 3.0 in stage 662.0 (TID 5299, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 4.0 in stage 662.0 (TID 5300, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 5.0 in stage 662.0 (TID 5301, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 6.0 in stage 662.0 (TID 5302, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 7.0 in stage 662.0 (TID 5303, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:35 INFO Executor: Running task 2.0 in stage 662.0 (TID 5298)
16/09/07 19:10:35 INFO Executor: Running task 3.0 in stage 662.0 (TID 5299)
16/09/07 19:10:35 INFO Executor: Running task 4.0 in stage 662.0 (TID 5300)
16/09/07 19:10:35 INFO Executor: Running task 5.0 in stage 662.0 (TID 5301)
16/09/07 19:10:35 INFO Executor: Running task 6.0 in stage 662.0 (TID 5302)
16/09/07 19:10:35 INFO Executor: Running task 7.0 in stage 662.0 (TID 5303)
16/09/07 19:10:35 INFO Executor: Running task 0.0 in stage 662.0 (TID 5296)
16/09/07 19:10:35 INFO Executor: Running task 1.0 in stage 662.0 (TID 5297)
16/09/07 19:10:35 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:35 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:35 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:35 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:35 INFO Executor: Finished task 2.0 in stage 662.0 (TID 5298). 2260 bytes result sent to driver
16/09/07 19:10:35 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:35 INFO Executor: Finished task 0.0 in stage 662.0 (TID 5296). 2260 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 2.0 in stage 662.0 (TID 5298) in 3 ms on localhost (1/8)
16/09/07 19:10:35 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:35 INFO TaskSetManager: Finished task 0.0 in stage 662.0 (TID 5296) in 3 ms on localhost (2/8)
16/09/07 19:10:35 INFO Executor: Finished task 1.0 in stage 662.0 (TID 5297). 2260 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 1.0 in stage 662.0 (TID 5297) in 3 ms on localhost (3/8)
16/09/07 19:10:35 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:35 INFO Executor: Finished task 5.0 in stage 662.0 (TID 5301). 2260 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 5.0 in stage 662.0 (TID 5301) in 3 ms on localhost (4/8)
16/09/07 19:10:35 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:35 INFO Executor: Finished task 7.0 in stage 662.0 (TID 5303). 2260 bytes result sent to driver
16/09/07 19:10:35 INFO Executor: Finished task 3.0 in stage 662.0 (TID 5299). 2260 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 7.0 in stage 662.0 (TID 5303) in 4 ms on localhost (5/8)
16/09/07 19:10:35 INFO TaskSetManager: Finished task 3.0 in stage 662.0 (TID 5299) in 4 ms on localhost (6/8)
16/09/07 19:10:35 INFO Executor: Finished task 4.0 in stage 662.0 (TID 5300). 2260 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 4.0 in stage 662.0 (TID 5300) in 4 ms on localhost (7/8)
16/09/07 19:10:35 INFO Executor: Finished task 6.0 in stage 662.0 (TID 5302). 2260 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 6.0 in stage 662.0 (TID 5302) in 4 ms on localhost (8/8)
16/09/07 19:10:35 INFO TaskSchedulerImpl: Removed TaskSet 662.0, whose tasks have all completed, from pool 
16/09/07 19:10:35 INFO DAGScheduler: ShuffleMapStage 662 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:35 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:35 INFO DAGScheduler: running: Set()
16/09/07 19:10:35 INFO DAGScheduler: waiting: Set(ResultStage 663)
16/09/07 19:10:35 INFO DAGScheduler: failed: Set()
16/09/07 19:10:35 INFO DAGScheduler: Missing parents for ResultStage 663: List()
16/09/07 19:10:35 INFO DAGScheduler: Submitting ResultStage 663 (MapPartitionsRDD[2042] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:35 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18935671, maxMem=4042889625
16/09/07 19:10:35 INFO MemoryStore: Block broadcast_1003 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:35 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18940767, maxMem=4042889625
16/09/07 19:10:35 INFO MemoryStore: Block broadcast_1003_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:35 INFO BlockManagerInfo: Added broadcast_1003_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:35 INFO SparkContext: Created broadcast 1003 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:35 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 663 (MapPartitionsRDD[2042] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:35 INFO TaskSchedulerImpl: Adding task set 663.0 with 8 tasks
16/09/07 19:10:35 INFO TaskSetManager: Starting task 0.0 in stage 663.0 (TID 5304, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 1.0 in stage 663.0 (TID 5305, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 2.0 in stage 663.0 (TID 5306, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 3.0 in stage 663.0 (TID 5307, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 4.0 in stage 663.0 (TID 5308, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 5.0 in stage 663.0 (TID 5309, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 6.0 in stage 663.0 (TID 5310, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:35 INFO TaskSetManager: Starting task 7.0 in stage 663.0 (TID 5311, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:35 INFO Executor: Running task 0.0 in stage 663.0 (TID 5304)
16/09/07 19:10:35 INFO Executor: Running task 3.0 in stage 663.0 (TID 5307)
16/09/07 19:10:35 INFO Executor: Running task 6.0 in stage 663.0 (TID 5310)
16/09/07 19:10:35 INFO Executor: Running task 1.0 in stage 663.0 (TID 5305)
16/09/07 19:10:35 INFO Executor: Running task 4.0 in stage 663.0 (TID 5308)
16/09/07 19:10:35 INFO Executor: Running task 5.0 in stage 663.0 (TID 5309)
16/09/07 19:10:35 INFO Executor: Running task 2.0 in stage 663.0 (TID 5306)
16/09/07 19:10:35 INFO Executor: Running task 7.0 in stage 663.0 (TID 5311)
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:35 INFO Executor: Finished task 4.0 in stage 663.0 (TID 5308). 2160 bytes result sent to driver
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:35 INFO TaskSetManager: Finished task 4.0 in stage 663.0 (TID 5308) in 2 ms on localhost (1/8)
16/09/07 19:10:35 INFO Executor: Finished task 2.0 in stage 663.0 (TID 5306). 2160 bytes result sent to driver
16/09/07 19:10:35 INFO Executor: Finished task 5.0 in stage 663.0 (TID 5309). 2160 bytes result sent to driver
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:35 INFO TaskSetManager: Finished task 2.0 in stage 663.0 (TID 5306) in 2 ms on localhost (2/8)
16/09/07 19:10:35 INFO TaskSetManager: Finished task 5.0 in stage 663.0 (TID 5309) in 2 ms on localhost (3/8)
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:35 INFO Executor: Finished task 3.0 in stage 663.0 (TID 5307). 2160 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 3.0 in stage 663.0 (TID 5307) in 3 ms on localhost (4/8)
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:35 INFO BlockManagerInfo: Removed broadcast_1002_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:35 INFO Executor: Finished task 1.0 in stage 663.0 (TID 5305). 3485274 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 1.0 in stage 663.0 (TID 5305) in 621 ms on localhost (5/8)
16/09/07 19:10:35 INFO Executor: Finished task 7.0 in stage 663.0 (TID 5311). 3485274 bytes result sent to driver
16/09/07 19:10:35 INFO TaskSetManager: Finished task 7.0 in stage 663.0 (TID 5311) in 912 ms on localhost (6/8)
16/09/07 19:10:36 INFO Executor: Finished task 0.0 in stage 663.0 (TID 5304). 3485274 bytes result sent to driver
16/09/07 19:10:36 INFO Executor: Finished task 6.0 in stage 663.0 (TID 5310). 3485274 bytes result sent to driver
16/09/07 19:10:36 INFO TaskSetManager: Finished task 0.0 in stage 663.0 (TID 5304) in 963 ms on localhost (7/8)
16/09/07 19:10:36 INFO TaskSetManager: Finished task 6.0 in stage 663.0 (TID 5310) in 966 ms on localhost (8/8)
16/09/07 19:10:36 INFO DAGScheduler: ResultStage 663 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.966 s
16/09/07 19:10:36 INFO TaskSchedulerImpl: Removed TaskSet 663.0, whose tasks have all completed, from pool 
16/09/07 19:10:36 INFO DAGScheduler: Job 336 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.974644 s
16/09/07 19:10:36 INFO ParameterAveragingTrainingMaster: Completed training of split 15 of 78
16/09/07 19:10:36 INFO ParameterAveragingTrainingMaster: Starting training of split 16 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:36 INFO MemoryStore: ensureFreeSpace(18123784) called with curMem=18936482, maxMem=4042889625
16/09/07 19:10:36 INFO MemoryStore: Block broadcast_1004 stored as values in memory (estimated size 17.3 MB, free 3.7 GB)
16/09/07 19:10:36 INFO BlockManagerInfo: Removed broadcast_1003_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:36 INFO ContextCleaner: Cleaned accumulator 664
16/09/07 19:10:36 INFO ContextCleaner: Cleaned accumulator 663
16/09/07 19:10:36 INFO BlockManagerInfo: Removed broadcast_1001_piece0 on localhost:41251 in memory (size: 269.7 KB, free: 3.8 GB)
16/09/07 19:10:36 INFO ContextCleaner: Cleaned shuffle 326
16/09/07 19:10:36 INFO MemoryStore: ensureFreeSpace(276451) called with curMem=18667608, maxMem=4042889625
16/09/07 19:10:36 INFO MemoryStore: Block broadcast_1004_piece0 stored as bytes in memory (estimated size 270.0 KB, free 3.7 GB)
16/09/07 19:10:36 INFO BlockManagerInfo: Added broadcast_1004_piece0 in memory on localhost:41251 (size: 270.0 KB, free: 3.8 GB)
16/09/07 19:10:36 INFO SparkContext: Created broadcast 1004 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:36 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:36 INFO DAGScheduler: Registering RDD 2043 (repartition at SparkUtils.java:190)
16/09/07 19:10:36 INFO DAGScheduler: Got job 337 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:36 INFO DAGScheduler: Final stage: ResultStage 665(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:36 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 664)
16/09/07 19:10:36 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 664)
16/09/07 19:10:36 INFO DAGScheduler: Submitting ShuffleMapStage 664 (MapPartitionsRDD[2043] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:36 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18944059, maxMem=4042889625
16/09/07 19:10:36 INFO MemoryStore: Block broadcast_1005 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:36 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18948523, maxMem=4042889625
16/09/07 19:10:36 INFO MemoryStore: Block broadcast_1005_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:36 INFO BlockManagerInfo: Added broadcast_1005_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:36 INFO SparkContext: Created broadcast 1005 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:36 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 664 (MapPartitionsRDD[2043] at repartition at SparkUtils.java:190)
16/09/07 19:10:36 INFO TaskSchedulerImpl: Adding task set 664.0 with 8 tasks
16/09/07 19:10:36 INFO TaskSetManager: Starting task 0.0 in stage 664.0 (TID 5312, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 1.0 in stage 664.0 (TID 5313, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 2.0 in stage 664.0 (TID 5314, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 3.0 in stage 664.0 (TID 5315, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 4.0 in stage 664.0 (TID 5316, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 5.0 in stage 664.0 (TID 5317, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 6.0 in stage 664.0 (TID 5318, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 7.0 in stage 664.0 (TID 5319, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:36 INFO Executor: Running task 2.0 in stage 664.0 (TID 5314)
16/09/07 19:10:36 INFO Executor: Running task 1.0 in stage 664.0 (TID 5313)
16/09/07 19:10:36 INFO Executor: Running task 5.0 in stage 664.0 (TID 5317)
16/09/07 19:10:36 INFO Executor: Running task 4.0 in stage 664.0 (TID 5316)
16/09/07 19:10:36 INFO Executor: Running task 7.0 in stage 664.0 (TID 5319)
16/09/07 19:10:36 INFO Executor: Running task 3.0 in stage 664.0 (TID 5315)
16/09/07 19:10:36 INFO Executor: Running task 0.0 in stage 664.0 (TID 5312)
16/09/07 19:10:36 INFO Executor: Running task 6.0 in stage 664.0 (TID 5318)
16/09/07 19:10:36 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:36 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:36 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:36 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:36 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:36 INFO Executor: Finished task 4.0 in stage 664.0 (TID 5316). 2260 bytes result sent to driver
16/09/07 19:10:36 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:36 INFO TaskSetManager: Finished task 4.0 in stage 664.0 (TID 5316) in 2 ms on localhost (1/8)
16/09/07 19:10:36 INFO Executor: Finished task 6.0 in stage 664.0 (TID 5318). 2260 bytes result sent to driver
16/09/07 19:10:36 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:36 INFO Executor: Finished task 5.0 in stage 664.0 (TID 5317). 2260 bytes result sent to driver
16/09/07 19:10:36 INFO TaskSetManager: Finished task 6.0 in stage 664.0 (TID 5318) in 2 ms on localhost (2/8)
16/09/07 19:10:36 INFO TaskSetManager: Finished task 5.0 in stage 664.0 (TID 5317) in 3 ms on localhost (3/8)
16/09/07 19:10:36 INFO Executor: Finished task 7.0 in stage 664.0 (TID 5319). 2260 bytes result sent to driver
16/09/07 19:10:36 INFO TaskSetManager: Finished task 7.0 in stage 664.0 (TID 5319) in 3 ms on localhost (4/8)
16/09/07 19:10:36 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:36 INFO Executor: Finished task 0.0 in stage 664.0 (TID 5312). 2260 bytes result sent to driver
16/09/07 19:10:36 INFO TaskSetManager: Finished task 0.0 in stage 664.0 (TID 5312) in 3 ms on localhost (5/8)
16/09/07 19:10:36 INFO Executor: Finished task 3.0 in stage 664.0 (TID 5315). 2260 bytes result sent to driver
16/09/07 19:10:36 INFO Executor: Finished task 2.0 in stage 664.0 (TID 5314). 2260 bytes result sent to driver
16/09/07 19:10:36 INFO Executor: Finished task 1.0 in stage 664.0 (TID 5313). 2260 bytes result sent to driver
16/09/07 19:10:36 INFO TaskSetManager: Finished task 3.0 in stage 664.0 (TID 5315) in 4 ms on localhost (6/8)
16/09/07 19:10:36 INFO TaskSetManager: Finished task 2.0 in stage 664.0 (TID 5314) in 4 ms on localhost (7/8)
16/09/07 19:10:36 INFO TaskSetManager: Finished task 1.0 in stage 664.0 (TID 5313) in 4 ms on localhost (8/8)
16/09/07 19:10:36 INFO TaskSchedulerImpl: Removed TaskSet 664.0, whose tasks have all completed, from pool 
16/09/07 19:10:36 INFO DAGScheduler: ShuffleMapStage 664 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:36 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:36 INFO DAGScheduler: running: Set()
16/09/07 19:10:36 INFO DAGScheduler: waiting: Set(ResultStage 665)
16/09/07 19:10:36 INFO DAGScheduler: failed: Set()
16/09/07 19:10:36 INFO DAGScheduler: Missing parents for ResultStage 665: List()
16/09/07 19:10:36 INFO DAGScheduler: Submitting ResultStage 665 (MapPartitionsRDD[2047] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:36 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18951087, maxMem=4042889625
16/09/07 19:10:36 INFO MemoryStore: Block broadcast_1006 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:36 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18956183, maxMem=4042889625
16/09/07 19:10:36 INFO MemoryStore: Block broadcast_1006_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:36 INFO BlockManagerInfo: Added broadcast_1006_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:36 INFO SparkContext: Created broadcast 1006 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:36 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 665 (MapPartitionsRDD[2047] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:36 INFO TaskSchedulerImpl: Adding task set 665.0 with 8 tasks
16/09/07 19:10:36 INFO TaskSetManager: Starting task 0.0 in stage 665.0 (TID 5320, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 1.0 in stage 665.0 (TID 5321, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 2.0 in stage 665.0 (TID 5322, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 3.0 in stage 665.0 (TID 5323, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 4.0 in stage 665.0 (TID 5324, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 5.0 in stage 665.0 (TID 5325, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 6.0 in stage 665.0 (TID 5326, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:36 INFO TaskSetManager: Starting task 7.0 in stage 665.0 (TID 5327, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:36 INFO Executor: Running task 1.0 in stage 665.0 (TID 5321)
16/09/07 19:10:36 INFO Executor: Running task 5.0 in stage 665.0 (TID 5325)
16/09/07 19:10:36 INFO Executor: Running task 6.0 in stage 665.0 (TID 5326)
16/09/07 19:10:36 INFO Executor: Running task 4.0 in stage 665.0 (TID 5324)
16/09/07 19:10:36 INFO Executor: Running task 3.0 in stage 665.0 (TID 5323)
16/09/07 19:10:36 INFO Executor: Running task 0.0 in stage 665.0 (TID 5320)
16/09/07 19:10:36 INFO Executor: Running task 2.0 in stage 665.0 (TID 5322)
16/09/07 19:10:36 INFO Executor: Running task 7.0 in stage 665.0 (TID 5327)
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:36 INFO Executor: Finished task 5.0 in stage 665.0 (TID 5325). 2160 bytes result sent to driver
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:36 INFO TaskSetManager: Finished task 5.0 in stage 665.0 (TID 5325) in 1 ms on localhost (1/8)
16/09/07 19:10:36 INFO Executor: Finished task 2.0 in stage 665.0 (TID 5322). 2160 bytes result sent to driver
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:36 INFO TaskSetManager: Finished task 2.0 in stage 665.0 (TID 5322) in 2 ms on localhost (2/8)
16/09/07 19:10:36 INFO Executor: Finished task 1.0 in stage 665.0 (TID 5321). 2160 bytes result sent to driver
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:36 INFO TaskSetManager: Finished task 1.0 in stage 665.0 (TID 5321) in 3 ms on localhost (3/8)
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:36 INFO Executor: Finished task 3.0 in stage 665.0 (TID 5323). 2160 bytes result sent to driver
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:36 INFO TaskSetManager: Finished task 3.0 in stage 665.0 (TID 5323) in 3 ms on localhost (4/8)
16/09/07 19:10:36 INFO Executor: Finished task 4.0 in stage 665.0 (TID 5324). 2160 bytes result sent to driver
16/09/07 19:10:36 INFO TaskSetManager: Finished task 4.0 in stage 665.0 (TID 5324) in 3 ms on localhost (5/8)
16/09/07 19:10:36 INFO BlockManagerInfo: Removed broadcast_1005_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:37 INFO Executor: Finished task 0.0 in stage 665.0 (TID 5320). 3485274 bytes result sent to driver
16/09/07 19:10:37 INFO TaskSetManager: Finished task 0.0 in stage 665.0 (TID 5320) in 873 ms on localhost (6/8)
16/09/07 19:10:37 INFO Executor: Finished task 7.0 in stage 665.0 (TID 5327). 3485274 bytes result sent to driver
16/09/07 19:10:37 INFO Executor: Finished task 6.0 in stage 665.0 (TID 5326). 3485274 bytes result sent to driver
16/09/07 19:10:37 INFO TaskSetManager: Finished task 6.0 in stage 665.0 (TID 5326) in 1035 ms on localhost (7/8)
16/09/07 19:10:37 INFO TaskSetManager: Finished task 7.0 in stage 665.0 (TID 5327) in 1035 ms on localhost (8/8)
16/09/07 19:10:37 INFO TaskSchedulerImpl: Removed TaskSet 665.0, whose tasks have all completed, from pool 
16/09/07 19:10:37 INFO DAGScheduler: ResultStage 665 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 1.036 s
16/09/07 19:10:37 INFO DAGScheduler: Job 337 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.044084 s
16/09/07 19:10:37 INFO ParameterAveragingTrainingMaster: Completed training of split 16 of 78
16/09/07 19:10:37 INFO ParameterAveragingTrainingMaster: Starting training of split 17 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:37 INFO MemoryStore: ensureFreeSpace(18168056) called with curMem=18951898, maxMem=4042889625
16/09/07 19:10:37 INFO MemoryStore: Block broadcast_1007 stored as values in memory (estimated size 17.3 MB, free 3.7 GB)
16/09/07 19:10:37 INFO BlockManagerInfo: Removed broadcast_1006_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:37 INFO ContextCleaner: Cleaned accumulator 666
16/09/07 19:10:37 INFO ContextCleaner: Cleaned accumulator 665
16/09/07 19:10:37 INFO BlockManagerInfo: Removed broadcast_1004_piece0 on localhost:41251 in memory (size: 270.0 KB, free: 3.8 GB)
16/09/07 19:10:37 INFO ContextCleaner: Cleaned shuffle 327
16/09/07 19:10:37 INFO MemoryStore: ensureFreeSpace(276577) called with curMem=18711880, maxMem=4042889625
16/09/07 19:10:37 INFO MemoryStore: Block broadcast_1007_piece0 stored as bytes in memory (estimated size 270.1 KB, free 3.7 GB)
16/09/07 19:10:37 INFO BlockManagerInfo: Added broadcast_1007_piece0 in memory on localhost:41251 (size: 270.1 KB, free: 3.8 GB)
16/09/07 19:10:37 INFO SparkContext: Created broadcast 1007 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:37 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:37 INFO DAGScheduler: Registering RDD 2048 (repartition at SparkUtils.java:190)
16/09/07 19:10:37 INFO DAGScheduler: Got job 338 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:37 INFO DAGScheduler: Final stage: ResultStage 667(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:37 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 666)
16/09/07 19:10:37 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 666)
16/09/07 19:10:37 INFO DAGScheduler: Submitting ShuffleMapStage 666 (MapPartitionsRDD[2048] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:37 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18988457, maxMem=4042889625
16/09/07 19:10:37 INFO MemoryStore: Block broadcast_1008 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:37 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18992921, maxMem=4042889625
16/09/07 19:10:37 INFO MemoryStore: Block broadcast_1008_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:37 INFO BlockManagerInfo: Added broadcast_1008_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:37 INFO SparkContext: Created broadcast 1008 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:37 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 666 (MapPartitionsRDD[2048] at repartition at SparkUtils.java:190)
16/09/07 19:10:37 INFO TaskSchedulerImpl: Adding task set 666.0 with 8 tasks
16/09/07 19:10:37 INFO TaskSetManager: Starting task 0.0 in stage 666.0 (TID 5328, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 1.0 in stage 666.0 (TID 5329, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 2.0 in stage 666.0 (TID 5330, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 3.0 in stage 666.0 (TID 5331, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 4.0 in stage 666.0 (TID 5332, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 5.0 in stage 666.0 (TID 5333, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 6.0 in stage 666.0 (TID 5334, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 7.0 in stage 666.0 (TID 5335, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:37 INFO Executor: Running task 0.0 in stage 666.0 (TID 5328)
16/09/07 19:10:37 INFO Executor: Running task 2.0 in stage 666.0 (TID 5330)
16/09/07 19:10:37 INFO Executor: Running task 7.0 in stage 666.0 (TID 5335)
16/09/07 19:10:37 INFO Executor: Running task 3.0 in stage 666.0 (TID 5331)
16/09/07 19:10:37 INFO Executor: Running task 5.0 in stage 666.0 (TID 5333)
16/09/07 19:10:37 INFO Executor: Running task 6.0 in stage 666.0 (TID 5334)
16/09/07 19:10:37 INFO Executor: Running task 4.0 in stage 666.0 (TID 5332)
16/09/07 19:10:37 INFO Executor: Running task 1.0 in stage 666.0 (TID 5329)
16/09/07 19:10:37 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:37 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:37 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:37 INFO Executor: Finished task 5.0 in stage 666.0 (TID 5333). 2260 bytes result sent to driver
16/09/07 19:10:37 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:37 INFO TaskSetManager: Finished task 5.0 in stage 666.0 (TID 5333) in 2 ms on localhost (1/8)
16/09/07 19:10:37 INFO Executor: Finished task 1.0 in stage 666.0 (TID 5329). 2260 bytes result sent to driver
16/09/07 19:10:37 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:37 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:37 INFO TaskSetManager: Finished task 1.0 in stage 666.0 (TID 5329) in 3 ms on localhost (2/8)
16/09/07 19:10:37 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:37 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:37 INFO Executor: Finished task 3.0 in stage 666.0 (TID 5331). 2260 bytes result sent to driver
16/09/07 19:10:37 INFO Executor: Finished task 6.0 in stage 666.0 (TID 5334). 2260 bytes result sent to driver
16/09/07 19:10:37 INFO Executor: Finished task 0.0 in stage 666.0 (TID 5328). 2260 bytes result sent to driver
16/09/07 19:10:37 INFO TaskSetManager: Finished task 3.0 in stage 666.0 (TID 5331) in 4 ms on localhost (3/8)
16/09/07 19:10:37 INFO Executor: Finished task 2.0 in stage 666.0 (TID 5330). 2260 bytes result sent to driver
16/09/07 19:10:37 INFO TaskSetManager: Finished task 6.0 in stage 666.0 (TID 5334) in 4 ms on localhost (4/8)
16/09/07 19:10:37 INFO TaskSetManager: Finished task 0.0 in stage 666.0 (TID 5328) in 4 ms on localhost (5/8)
16/09/07 19:10:37 INFO TaskSetManager: Finished task 2.0 in stage 666.0 (TID 5330) in 4 ms on localhost (6/8)
16/09/07 19:10:37 INFO Executor: Finished task 7.0 in stage 666.0 (TID 5335). 2260 bytes result sent to driver
16/09/07 19:10:37 INFO TaskSetManager: Finished task 7.0 in stage 666.0 (TID 5335) in 4 ms on localhost (7/8)
16/09/07 19:10:37 INFO Executor: Finished task 4.0 in stage 666.0 (TID 5332). 2260 bytes result sent to driver
16/09/07 19:10:37 INFO TaskSetManager: Finished task 4.0 in stage 666.0 (TID 5332) in 4 ms on localhost (8/8)
16/09/07 19:10:37 INFO TaskSchedulerImpl: Removed TaskSet 666.0, whose tasks have all completed, from pool 
16/09/07 19:10:37 INFO DAGScheduler: ShuffleMapStage 666 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:37 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:37 INFO DAGScheduler: running: Set()
16/09/07 19:10:37 INFO DAGScheduler: waiting: Set(ResultStage 667)
16/09/07 19:10:37 INFO DAGScheduler: failed: Set()
16/09/07 19:10:37 INFO DAGScheduler: Missing parents for ResultStage 667: List()
16/09/07 19:10:37 INFO DAGScheduler: Submitting ResultStage 667 (MapPartitionsRDD[2052] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:37 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18995485, maxMem=4042889625
16/09/07 19:10:37 INFO MemoryStore: Block broadcast_1009 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:37 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19000581, maxMem=4042889625
16/09/07 19:10:37 INFO MemoryStore: Block broadcast_1009_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:37 INFO BlockManagerInfo: Added broadcast_1009_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:37 INFO SparkContext: Created broadcast 1009 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:37 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 667 (MapPartitionsRDD[2052] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:37 INFO TaskSchedulerImpl: Adding task set 667.0 with 8 tasks
16/09/07 19:10:37 INFO TaskSetManager: Starting task 0.0 in stage 667.0 (TID 5336, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 1.0 in stage 667.0 (TID 5337, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 2.0 in stage 667.0 (TID 5338, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 3.0 in stage 667.0 (TID 5339, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 4.0 in stage 667.0 (TID 5340, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 5.0 in stage 667.0 (TID 5341, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 6.0 in stage 667.0 (TID 5342, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:37 INFO TaskSetManager: Starting task 7.0 in stage 667.0 (TID 5343, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:37 INFO Executor: Running task 0.0 in stage 667.0 (TID 5336)
16/09/07 19:10:37 INFO Executor: Running task 2.0 in stage 667.0 (TID 5338)
16/09/07 19:10:37 INFO Executor: Running task 6.0 in stage 667.0 (TID 5342)
16/09/07 19:10:37 INFO Executor: Running task 3.0 in stage 667.0 (TID 5339)
16/09/07 19:10:37 INFO Executor: Running task 5.0 in stage 667.0 (TID 5341)
16/09/07 19:10:37 INFO Executor: Running task 1.0 in stage 667.0 (TID 5337)
16/09/07 19:10:37 INFO Executor: Running task 7.0 in stage 667.0 (TID 5343)
16/09/07 19:10:37 INFO Executor: Running task 4.0 in stage 667.0 (TID 5340)
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:37 INFO Executor: Finished task 4.0 in stage 667.0 (TID 5340). 2160 bytes result sent to driver
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:37 INFO Executor: Finished task 1.0 in stage 667.0 (TID 5337). 2160 bytes result sent to driver
16/09/07 19:10:37 INFO TaskSetManager: Finished task 4.0 in stage 667.0 (TID 5340) in 2 ms on localhost (1/8)
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:37 INFO Executor: Finished task 5.0 in stage 667.0 (TID 5341). 2160 bytes result sent to driver
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:37 INFO Executor: Finished task 3.0 in stage 667.0 (TID 5339). 2160 bytes result sent to driver
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:37 INFO TaskSetManager: Finished task 1.0 in stage 667.0 (TID 5337) in 4 ms on localhost (2/8)
16/09/07 19:10:37 INFO TaskSetManager: Finished task 5.0 in stage 667.0 (TID 5341) in 4 ms on localhost (3/8)
16/09/07 19:10:37 INFO TaskSetManager: Finished task 3.0 in stage 667.0 (TID 5339) in 5 ms on localhost (4/8)
16/09/07 19:10:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:37 INFO Executor: Finished task 2.0 in stage 667.0 (TID 5338). 2160 bytes result sent to driver
16/09/07 19:10:37 INFO TaskSetManager: Finished task 2.0 in stage 667.0 (TID 5338) in 5 ms on localhost (5/8)
16/09/07 19:10:37 INFO BlockManagerInfo: Removed broadcast_1008_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:38 INFO Executor: Finished task 7.0 in stage 667.0 (TID 5343). 3485274 bytes result sent to driver
16/09/07 19:10:38 INFO Executor: Finished task 0.0 in stage 667.0 (TID 5336). 3485274 bytes result sent to driver
16/09/07 19:10:38 INFO Executor: Finished task 6.0 in stage 667.0 (TID 5342). 3485274 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 6.0 in stage 667.0 (TID 5342) in 871 ms on localhost (6/8)
16/09/07 19:10:38 INFO TaskSetManager: Finished task 7.0 in stage 667.0 (TID 5343) in 883 ms on localhost (7/8)
16/09/07 19:10:38 INFO TaskSetManager: Finished task 0.0 in stage 667.0 (TID 5336) in 885 ms on localhost (8/8)
16/09/07 19:10:38 INFO TaskSchedulerImpl: Removed TaskSet 667.0, whose tasks have all completed, from pool 
16/09/07 19:10:38 INFO DAGScheduler: ResultStage 667 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.885 s
16/09/07 19:10:38 INFO DAGScheduler: Job 338 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.893550 s
16/09/07 19:10:38 INFO ParameterAveragingTrainingMaster: Completed training of split 17 of 78
16/09/07 19:10:38 INFO ParameterAveragingTrainingMaster: Starting training of split 18 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:38 INFO MemoryStore: ensureFreeSpace(18191328) called with curMem=18996296, maxMem=4042889625
16/09/07 19:10:38 INFO MemoryStore: Block broadcast_1010 stored as values in memory (estimated size 17.3 MB, free 3.7 GB)
16/09/07 19:10:38 INFO MemoryStore: ensureFreeSpace(276329) called with curMem=37187624, maxMem=4042889625
16/09/07 19:10:38 INFO MemoryStore: Block broadcast_1010_piece0 stored as bytes in memory (estimated size 269.9 KB, free 3.7 GB)
16/09/07 19:10:38 INFO BlockManagerInfo: Added broadcast_1010_piece0 in memory on localhost:41251 (size: 269.9 KB, free: 3.8 GB)
16/09/07 19:10:38 INFO SparkContext: Created broadcast 1010 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:38 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:38 INFO DAGScheduler: Registering RDD 2053 (repartition at SparkUtils.java:190)
16/09/07 19:10:38 INFO DAGScheduler: Got job 339 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:38 INFO DAGScheduler: Final stage: ResultStage 669(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:38 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 668)
16/09/07 19:10:38 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 668)
16/09/07 19:10:38 INFO DAGScheduler: Submitting ShuffleMapStage 668 (MapPartitionsRDD[2053] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:38 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=37463953, maxMem=4042889625
16/09/07 19:10:38 INFO MemoryStore: Block broadcast_1011 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:38 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=37468417, maxMem=4042889625
16/09/07 19:10:38 INFO MemoryStore: Block broadcast_1011_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:38 INFO BlockManagerInfo: Added broadcast_1011_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:38 INFO SparkContext: Created broadcast 1011 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:38 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 668 (MapPartitionsRDD[2053] at repartition at SparkUtils.java:190)
16/09/07 19:10:38 INFO TaskSchedulerImpl: Adding task set 668.0 with 8 tasks
16/09/07 19:10:38 INFO TaskSetManager: Starting task 0.0 in stage 668.0 (TID 5344, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 1.0 in stage 668.0 (TID 5345, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 2.0 in stage 668.0 (TID 5346, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 3.0 in stage 668.0 (TID 5347, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 4.0 in stage 668.0 (TID 5348, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 5.0 in stage 668.0 (TID 5349, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 6.0 in stage 668.0 (TID 5350, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 7.0 in stage 668.0 (TID 5351, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:38 INFO Executor: Running task 2.0 in stage 668.0 (TID 5346)
16/09/07 19:10:38 INFO Executor: Running task 1.0 in stage 668.0 (TID 5345)
16/09/07 19:10:38 INFO Executor: Running task 3.0 in stage 668.0 (TID 5347)
16/09/07 19:10:38 INFO Executor: Running task 6.0 in stage 668.0 (TID 5350)
16/09/07 19:10:38 INFO Executor: Running task 4.0 in stage 668.0 (TID 5348)
16/09/07 19:10:38 INFO Executor: Running task 0.0 in stage 668.0 (TID 5344)
16/09/07 19:10:38 INFO Executor: Running task 7.0 in stage 668.0 (TID 5351)
16/09/07 19:10:38 INFO Executor: Running task 5.0 in stage 668.0 (TID 5349)
16/09/07 19:10:38 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:38 INFO BlockManagerInfo: Removed broadcast_1009_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:38 INFO ContextCleaner: Cleaned accumulator 668
16/09/07 19:10:38 INFO ContextCleaner: Cleaned accumulator 667
16/09/07 19:10:38 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:38 INFO BlockManagerInfo: Removed broadcast_1007_piece0 on localhost:41251 in memory (size: 270.1 KB, free: 3.8 GB)
16/09/07 19:10:38 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:38 INFO Executor: Finished task 2.0 in stage 668.0 (TID 5346). 2260 bytes result sent to driver
16/09/07 19:10:38 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:38 INFO ContextCleaner: Cleaned shuffle 328
16/09/07 19:10:38 INFO TaskSetManager: Finished task 2.0 in stage 668.0 (TID 5346) in 74 ms on localhost (1/8)
16/09/07 19:10:38 INFO Executor: Finished task 7.0 in stage 668.0 (TID 5351). 2260 bytes result sent to driver
16/09/07 19:10:38 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:38 INFO TaskSetManager: Finished task 7.0 in stage 668.0 (TID 5351) in 74 ms on localhost (2/8)
16/09/07 19:10:38 INFO Executor: Finished task 1.0 in stage 668.0 (TID 5345). 2260 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 1.0 in stage 668.0 (TID 5345) in 75 ms on localhost (3/8)
16/09/07 19:10:38 INFO Executor: Finished task 3.0 in stage 668.0 (TID 5347). 2260 bytes result sent to driver
16/09/07 19:10:38 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:38 INFO Executor: Finished task 0.0 in stage 668.0 (TID 5344). 2260 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 3.0 in stage 668.0 (TID 5347) in 75 ms on localhost (4/8)
16/09/07 19:10:38 INFO TaskSetManager: Finished task 0.0 in stage 668.0 (TID 5344) in 75 ms on localhost (5/8)
16/09/07 19:10:38 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:38 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:38 INFO Executor: Finished task 4.0 in stage 668.0 (TID 5348). 2260 bytes result sent to driver
16/09/07 19:10:38 INFO Executor: Finished task 6.0 in stage 668.0 (TID 5350). 2260 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 4.0 in stage 668.0 (TID 5348) in 76 ms on localhost (6/8)
16/09/07 19:10:38 INFO Executor: Finished task 5.0 in stage 668.0 (TID 5349). 2260 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 6.0 in stage 668.0 (TID 5350) in 76 ms on localhost (7/8)
16/09/07 19:10:38 INFO TaskSetManager: Finished task 5.0 in stage 668.0 (TID 5349) in 76 ms on localhost (8/8)
16/09/07 19:10:38 INFO TaskSchedulerImpl: Removed TaskSet 668.0, whose tasks have all completed, from pool 
16/09/07 19:10:38 INFO DAGScheduler: ShuffleMapStage 668 (repartition at SparkUtils.java:190) finished in 0.076 s
16/09/07 19:10:38 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:38 INFO DAGScheduler: running: Set()
16/09/07 19:10:38 INFO DAGScheduler: waiting: Set(ResultStage 669)
16/09/07 19:10:38 INFO DAGScheduler: failed: Set()
16/09/07 19:10:38 INFO DAGScheduler: Missing parents for ResultStage 669: List()
16/09/07 19:10:38 INFO DAGScheduler: Submitting ResultStage 669 (MapPartitionsRDD[2057] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:38 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19018507, maxMem=4042889625
16/09/07 19:10:38 INFO MemoryStore: Block broadcast_1012 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:38 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=19023603, maxMem=4042889625
16/09/07 19:10:38 INFO MemoryStore: Block broadcast_1012_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:38 INFO BlockManagerInfo: Added broadcast_1012_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:38 INFO SparkContext: Created broadcast 1012 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:38 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 669 (MapPartitionsRDD[2057] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:38 INFO TaskSchedulerImpl: Adding task set 669.0 with 8 tasks
16/09/07 19:10:38 INFO TaskSetManager: Starting task 0.0 in stage 669.0 (TID 5352, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 1.0 in stage 669.0 (TID 5353, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 2.0 in stage 669.0 (TID 5354, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 3.0 in stage 669.0 (TID 5355, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 4.0 in stage 669.0 (TID 5356, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 5.0 in stage 669.0 (TID 5357, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 6.0 in stage 669.0 (TID 5358, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:38 INFO TaskSetManager: Starting task 7.0 in stage 669.0 (TID 5359, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:38 INFO Executor: Running task 2.0 in stage 669.0 (TID 5354)
16/09/07 19:10:38 INFO Executor: Running task 7.0 in stage 669.0 (TID 5359)
16/09/07 19:10:38 INFO Executor: Running task 5.0 in stage 669.0 (TID 5357)
16/09/07 19:10:38 INFO Executor: Running task 0.0 in stage 669.0 (TID 5352)
16/09/07 19:10:38 INFO Executor: Running task 6.0 in stage 669.0 (TID 5358)
16/09/07 19:10:38 INFO Executor: Running task 1.0 in stage 669.0 (TID 5353)
16/09/07 19:10:38 INFO Executor: Running task 4.0 in stage 669.0 (TID 5356)
16/09/07 19:10:38 INFO Executor: Running task 3.0 in stage 669.0 (TID 5355)
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:38 INFO Executor: Finished task 5.0 in stage 669.0 (TID 5357). 2160 bytes result sent to driver
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:38 INFO Executor: Finished task 4.0 in stage 669.0 (TID 5356). 2160 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 5.0 in stage 669.0 (TID 5357) in 2 ms on localhost (1/8)
16/09/07 19:10:38 INFO Executor: Finished task 3.0 in stage 669.0 (TID 5355). 2160 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 4.0 in stage 669.0 (TID 5356) in 2 ms on localhost (2/8)
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:38 INFO Executor: Finished task 1.0 in stage 669.0 (TID 5353). 2160 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 3.0 in stage 669.0 (TID 5355) in 2 ms on localhost (3/8)
16/09/07 19:10:38 INFO TaskSetManager: Finished task 1.0 in stage 669.0 (TID 5353) in 2 ms on localhost (4/8)
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:38 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:38 INFO Executor: Finished task 2.0 in stage 669.0 (TID 5354). 2160 bytes result sent to driver
16/09/07 19:10:38 INFO TaskSetManager: Finished task 2.0 in stage 669.0 (TID 5354) in 4 ms on localhost (5/8)
16/09/07 19:10:39 INFO BlockManagerInfo: Removed broadcast_1011_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:39 INFO Executor: Finished task 6.0 in stage 669.0 (TID 5358). 3485274 bytes result sent to driver
16/09/07 19:10:39 INFO Executor: Finished task 0.0 in stage 669.0 (TID 5352). 3485274 bytes result sent to driver
16/09/07 19:10:39 INFO Executor: Finished task 7.0 in stage 669.0 (TID 5359). 3485274 bytes result sent to driver
16/09/07 19:10:39 INFO TaskSetManager: Finished task 0.0 in stage 669.0 (TID 5352) in 823 ms on localhost (6/8)
16/09/07 19:10:39 INFO TaskSetManager: Finished task 6.0 in stage 669.0 (TID 5358) in 838 ms on localhost (7/8)
16/09/07 19:10:39 INFO TaskSetManager: Finished task 7.0 in stage 669.0 (TID 5359) in 838 ms on localhost (8/8)
16/09/07 19:10:39 INFO TaskSchedulerImpl: Removed TaskSet 669.0, whose tasks have all completed, from pool 
16/09/07 19:10:39 INFO DAGScheduler: ResultStage 669 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.838 s
16/09/07 19:10:39 INFO DAGScheduler: Job 339 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.918745 s
16/09/07 19:10:39 INFO ParameterAveragingTrainingMaster: Completed training of split 18 of 78
16/09/07 19:10:39 INFO ParameterAveragingTrainingMaster: Starting training of split 19 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:39 INFO MemoryStore: ensureFreeSpace(18205480) called with curMem=19019321, maxMem=4042889625
16/09/07 19:10:39 INFO MemoryStore: Block broadcast_1013 stored as values in memory (estimated size 17.4 MB, free 3.7 GB)
16/09/07 19:10:39 INFO MemoryStore: ensureFreeSpace(276557) called with curMem=37224801, maxMem=4042889625
16/09/07 19:10:39 INFO MemoryStore: Block broadcast_1013_piece0 stored as bytes in memory (estimated size 270.1 KB, free 3.7 GB)
16/09/07 19:10:39 INFO BlockManagerInfo: Added broadcast_1013_piece0 in memory on localhost:41251 (size: 270.1 KB, free: 3.8 GB)
16/09/07 19:10:39 INFO SparkContext: Created broadcast 1013 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:39 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:39 INFO DAGScheduler: Registering RDD 2058 (repartition at SparkUtils.java:190)
16/09/07 19:10:39 INFO DAGScheduler: Got job 340 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:39 INFO DAGScheduler: Final stage: ResultStage 671(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:39 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 670)
16/09/07 19:10:39 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 670)
16/09/07 19:10:39 INFO DAGScheduler: Submitting ShuffleMapStage 670 (MapPartitionsRDD[2058] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:39 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=37501358, maxMem=4042889625
16/09/07 19:10:39 INFO MemoryStore: Block broadcast_1014 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:39 INFO BlockManagerInfo: Removed broadcast_1012_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:39 INFO ContextCleaner: Cleaned accumulator 670
16/09/07 19:10:39 INFO ContextCleaner: Cleaned accumulator 669
16/09/07 19:10:39 INFO BlockManagerInfo: Removed broadcast_1010_piece0 on localhost:41251 in memory (size: 269.9 KB, free: 3.8 GB)
16/09/07 19:10:39 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=19030325, maxMem=4042889625
16/09/07 19:10:39 INFO MemoryStore: Block broadcast_1014_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:39 INFO ContextCleaner: Cleaned shuffle 329
16/09/07 19:10:39 INFO BlockManagerInfo: Added broadcast_1014_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:39 INFO SparkContext: Created broadcast 1014 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:39 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 670 (MapPartitionsRDD[2058] at repartition at SparkUtils.java:190)
16/09/07 19:10:39 INFO TaskSchedulerImpl: Adding task set 670.0 with 8 tasks
16/09/07 19:10:39 INFO TaskSetManager: Starting task 0.0 in stage 670.0 (TID 5360, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 1.0 in stage 670.0 (TID 5361, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 2.0 in stage 670.0 (TID 5362, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 3.0 in stage 670.0 (TID 5363, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 4.0 in stage 670.0 (TID 5364, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 5.0 in stage 670.0 (TID 5365, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 6.0 in stage 670.0 (TID 5366, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 7.0 in stage 670.0 (TID 5367, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:39 INFO Executor: Running task 0.0 in stage 670.0 (TID 5360)
16/09/07 19:10:39 INFO Executor: Running task 6.0 in stage 670.0 (TID 5366)
16/09/07 19:10:39 INFO Executor: Running task 7.0 in stage 670.0 (TID 5367)
16/09/07 19:10:39 INFO Executor: Running task 1.0 in stage 670.0 (TID 5361)
16/09/07 19:10:39 INFO Executor: Running task 4.0 in stage 670.0 (TID 5364)
16/09/07 19:10:39 INFO Executor: Running task 3.0 in stage 670.0 (TID 5363)
16/09/07 19:10:39 INFO Executor: Running task 2.0 in stage 670.0 (TID 5362)
16/09/07 19:10:39 INFO Executor: Running task 5.0 in stage 670.0 (TID 5365)
16/09/07 19:10:39 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:39 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:39 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:39 INFO Executor: Finished task 5.0 in stage 670.0 (TID 5365). 2260 bytes result sent to driver
16/09/07 19:10:39 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:39 INFO Executor: Finished task 7.0 in stage 670.0 (TID 5367). 2260 bytes result sent to driver
16/09/07 19:10:39 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:39 INFO TaskSetManager: Finished task 7.0 in stage 670.0 (TID 5367) in 2 ms on localhost (1/8)
16/09/07 19:10:39 INFO Executor: Finished task 2.0 in stage 670.0 (TID 5362). 2260 bytes result sent to driver
16/09/07 19:10:39 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:39 INFO TaskSetManager: Finished task 2.0 in stage 670.0 (TID 5362) in 2 ms on localhost (2/8)
16/09/07 19:10:39 INFO Executor: Finished task 4.0 in stage 670.0 (TID 5364). 2260 bytes result sent to driver
16/09/07 19:10:39 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:39 INFO TaskSetManager: Finished task 4.0 in stage 670.0 (TID 5364) in 3 ms on localhost (3/8)
16/09/07 19:10:39 INFO Executor: Finished task 0.0 in stage 670.0 (TID 5360). 2260 bytes result sent to driver
16/09/07 19:10:39 INFO TaskSetManager: Finished task 0.0 in stage 670.0 (TID 5360) in 3 ms on localhost (4/8)
16/09/07 19:10:39 INFO Executor: Finished task 3.0 in stage 670.0 (TID 5363). 2260 bytes result sent to driver
16/09/07 19:10:39 INFO Executor: Finished task 6.0 in stage 670.0 (TID 5366). 2260 bytes result sent to driver
16/09/07 19:10:39 INFO TaskSetManager: Finished task 5.0 in stage 670.0 (TID 5365) in 4 ms on localhost (5/8)
16/09/07 19:10:39 INFO TaskSetManager: Finished task 3.0 in stage 670.0 (TID 5363) in 4 ms on localhost (6/8)
16/09/07 19:10:39 INFO TaskSetManager: Finished task 6.0 in stage 670.0 (TID 5366) in 4 ms on localhost (7/8)
16/09/07 19:10:39 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:39 INFO Executor: Finished task 1.0 in stage 670.0 (TID 5361). 2260 bytes result sent to driver
16/09/07 19:10:39 INFO TaskSetManager: Finished task 1.0 in stage 670.0 (TID 5361) in 5 ms on localhost (8/8)
16/09/07 19:10:39 INFO TaskSchedulerImpl: Removed TaskSet 670.0, whose tasks have all completed, from pool 
16/09/07 19:10:39 INFO DAGScheduler: ShuffleMapStage 670 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:39 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:39 INFO DAGScheduler: running: Set()
16/09/07 19:10:39 INFO DAGScheduler: waiting: Set(ResultStage 671)
16/09/07 19:10:39 INFO DAGScheduler: failed: Set()
16/09/07 19:10:39 INFO DAGScheduler: Missing parents for ResultStage 671: List()
16/09/07 19:10:39 INFO DAGScheduler: Submitting ResultStage 671 (MapPartitionsRDD[2062] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:39 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19032888, maxMem=4042889625
16/09/07 19:10:39 INFO MemoryStore: Block broadcast_1015 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:39 INFO MemoryStore: ensureFreeSpace(2745) called with curMem=19037984, maxMem=4042889625
16/09/07 19:10:39 INFO MemoryStore: Block broadcast_1015_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:39 INFO BlockManagerInfo: Added broadcast_1015_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:39 INFO SparkContext: Created broadcast 1015 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:39 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 671 (MapPartitionsRDD[2062] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:39 INFO TaskSchedulerImpl: Adding task set 671.0 with 8 tasks
16/09/07 19:10:39 INFO TaskSetManager: Starting task 0.0 in stage 671.0 (TID 5368, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 1.0 in stage 671.0 (TID 5369, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 2.0 in stage 671.0 (TID 5370, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 3.0 in stage 671.0 (TID 5371, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 4.0 in stage 671.0 (TID 5372, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 5.0 in stage 671.0 (TID 5373, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 6.0 in stage 671.0 (TID 5374, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:39 INFO TaskSetManager: Starting task 7.0 in stage 671.0 (TID 5375, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:39 INFO Executor: Running task 1.0 in stage 671.0 (TID 5369)
16/09/07 19:10:39 INFO Executor: Running task 2.0 in stage 671.0 (TID 5370)
16/09/07 19:10:39 INFO Executor: Running task 6.0 in stage 671.0 (TID 5374)
16/09/07 19:10:39 INFO Executor: Running task 5.0 in stage 671.0 (TID 5373)
16/09/07 19:10:39 INFO Executor: Running task 0.0 in stage 671.0 (TID 5368)
16/09/07 19:10:39 INFO Executor: Running task 4.0 in stage 671.0 (TID 5372)
16/09/07 19:10:39 INFO Executor: Running task 7.0 in stage 671.0 (TID 5375)
16/09/07 19:10:39 INFO Executor: Running task 3.0 in stage 671.0 (TID 5371)
16/09/07 19:10:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:39 INFO Executor: Finished task 1.0 in stage 671.0 (TID 5369). 2160 bytes result sent to driver
16/09/07 19:10:39 INFO Executor: Finished task 3.0 in stage 671.0 (TID 5371). 2160 bytes result sent to driver
16/09/07 19:10:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:39 INFO TaskSetManager: Finished task 1.0 in stage 671.0 (TID 5369) in 2 ms on localhost (1/8)
16/09/07 19:10:39 INFO TaskSetManager: Finished task 3.0 in stage 671.0 (TID 5371) in 1 ms on localhost (2/8)
16/09/07 19:10:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:40 INFO Executor: Finished task 4.0 in stage 671.0 (TID 5372). 2160 bytes result sent to driver
16/09/07 19:10:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:40 INFO Executor: Finished task 2.0 in stage 671.0 (TID 5370). 2160 bytes result sent to driver
16/09/07 19:10:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:40 INFO TaskSetManager: Finished task 4.0 in stage 671.0 (TID 5372) in 2 ms on localhost (3/8)
16/09/07 19:10:40 INFO TaskSetManager: Finished task 2.0 in stage 671.0 (TID 5370) in 2 ms on localhost (4/8)
16/09/07 19:10:40 INFO Executor: Finished task 5.0 in stage 671.0 (TID 5373). 2160 bytes result sent to driver
16/09/07 19:10:40 INFO TaskSetManager: Finished task 5.0 in stage 671.0 (TID 5373) in 3 ms on localhost (5/8)
16/09/07 19:10:40 INFO BlockManagerInfo: Removed broadcast_1014_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:40 INFO Executor: Finished task 0.0 in stage 671.0 (TID 5368). 3485274 bytes result sent to driver
16/09/07 19:10:40 INFO Executor: Finished task 7.0 in stage 671.0 (TID 5375). 3485274 bytes result sent to driver
16/09/07 19:10:40 INFO Executor: Finished task 6.0 in stage 671.0 (TID 5374). 3485274 bytes result sent to driver
16/09/07 19:10:40 INFO TaskSetManager: Finished task 6.0 in stage 671.0 (TID 5374) in 766 ms on localhost (6/8)
16/09/07 19:10:40 INFO TaskSetManager: Finished task 0.0 in stage 671.0 (TID 5368) in 778 ms on localhost (7/8)
16/09/07 19:10:40 INFO TaskSetManager: Finished task 7.0 in stage 671.0 (TID 5375) in 777 ms on localhost (8/8)
16/09/07 19:10:40 INFO TaskSchedulerImpl: Removed TaskSet 671.0, whose tasks have all completed, from pool 
16/09/07 19:10:40 INFO DAGScheduler: ResultStage 671 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.778 s
16/09/07 19:10:40 INFO DAGScheduler: Job 340 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.850158 s
16/09/07 19:10:40 INFO ParameterAveragingTrainingMaster: Completed training of split 19 of 78
16/09/07 19:10:40 INFO ParameterAveragingTrainingMaster: Starting training of split 20 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:40 INFO BlockManagerInfo: Removed broadcast_1015_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:40 INFO ContextCleaner: Cleaned accumulator 672
16/09/07 19:10:40 INFO ContextCleaner: Cleaned accumulator 671
16/09/07 19:10:40 INFO BlockManagerInfo: Removed broadcast_1013_piece0 on localhost:41251 in memory (size: 270.1 KB, free: 3.8 GB)
16/09/07 19:10:40 INFO ContextCleaner: Cleaned shuffle 330
16/09/07 19:10:40 INFO MemoryStore: ensureFreeSpace(18210280) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:40 INFO MemoryStore: Block broadcast_1016 stored as values in memory (estimated size 17.4 MB, free 3.7 GB)
16/09/07 19:10:41 INFO MemoryStore: ensureFreeSpace(276388) called with curMem=18754104, maxMem=4042889625
16/09/07 19:10:41 INFO MemoryStore: Block broadcast_1016_piece0 stored as bytes in memory (estimated size 269.9 KB, free 3.7 GB)
16/09/07 19:10:41 INFO BlockManagerInfo: Added broadcast_1016_piece0 in memory on localhost:41251 (size: 269.9 KB, free: 3.8 GB)
16/09/07 19:10:41 INFO SparkContext: Created broadcast 1016 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:41 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:41 INFO DAGScheduler: Registering RDD 2063 (repartition at SparkUtils.java:190)
16/09/07 19:10:41 INFO DAGScheduler: Got job 341 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:41 INFO DAGScheduler: Final stage: ResultStage 673(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:41 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 672)
16/09/07 19:10:41 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 672)
16/09/07 19:10:41 INFO DAGScheduler: Submitting ShuffleMapStage 672 (MapPartitionsRDD[2063] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:41 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19030492, maxMem=4042889625
16/09/07 19:10:41 INFO MemoryStore: Block broadcast_1017 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:41 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19034956, maxMem=4042889625
16/09/07 19:10:41 INFO MemoryStore: Block broadcast_1017_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:41 INFO BlockManagerInfo: Added broadcast_1017_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:41 INFO SparkContext: Created broadcast 1017 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:41 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 672 (MapPartitionsRDD[2063] at repartition at SparkUtils.java:190)
16/09/07 19:10:41 INFO TaskSchedulerImpl: Adding task set 672.0 with 8 tasks
16/09/07 19:10:41 INFO TaskSetManager: Starting task 0.0 in stage 672.0 (TID 5376, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 1.0 in stage 672.0 (TID 5377, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 2.0 in stage 672.0 (TID 5378, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 3.0 in stage 672.0 (TID 5379, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 4.0 in stage 672.0 (TID 5380, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 5.0 in stage 672.0 (TID 5381, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 6.0 in stage 672.0 (TID 5382, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 7.0 in stage 672.0 (TID 5383, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:41 INFO Executor: Running task 1.0 in stage 672.0 (TID 5377)
16/09/07 19:10:41 INFO Executor: Running task 2.0 in stage 672.0 (TID 5378)
16/09/07 19:10:41 INFO Executor: Running task 7.0 in stage 672.0 (TID 5383)
16/09/07 19:10:41 INFO Executor: Running task 5.0 in stage 672.0 (TID 5381)
16/09/07 19:10:41 INFO Executor: Running task 6.0 in stage 672.0 (TID 5382)
16/09/07 19:10:41 INFO Executor: Running task 4.0 in stage 672.0 (TID 5380)
16/09/07 19:10:41 INFO Executor: Running task 3.0 in stage 672.0 (TID 5379)
16/09/07 19:10:41 INFO Executor: Running task 0.0 in stage 672.0 (TID 5376)
16/09/07 19:10:41 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:41 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:41 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:41 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:41 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:41 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:41 INFO Executor: Finished task 5.0 in stage 672.0 (TID 5381). 2260 bytes result sent to driver
16/09/07 19:10:41 INFO Executor: Finished task 4.0 in stage 672.0 (TID 5380). 2260 bytes result sent to driver
16/09/07 19:10:41 INFO Executor: Finished task 3.0 in stage 672.0 (TID 5379). 2260 bytes result sent to driver
16/09/07 19:10:41 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:41 INFO TaskSetManager: Finished task 4.0 in stage 672.0 (TID 5380) in 3 ms on localhost (1/8)
16/09/07 19:10:41 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:41 INFO Executor: Finished task 2.0 in stage 672.0 (TID 5378). 2260 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 5.0 in stage 672.0 (TID 5381) in 3 ms on localhost (2/8)
16/09/07 19:10:41 INFO TaskSetManager: Finished task 3.0 in stage 672.0 (TID 5379) in 3 ms on localhost (3/8)
16/09/07 19:10:41 INFO Executor: Finished task 7.0 in stage 672.0 (TID 5383). 2260 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 2.0 in stage 672.0 (TID 5378) in 3 ms on localhost (4/8)
16/09/07 19:10:41 INFO TaskSetManager: Finished task 7.0 in stage 672.0 (TID 5383) in 3 ms on localhost (5/8)
16/09/07 19:10:41 INFO Executor: Finished task 6.0 in stage 672.0 (TID 5382). 2260 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 6.0 in stage 672.0 (TID 5382) in 3 ms on localhost (6/8)
16/09/07 19:10:41 INFO Executor: Finished task 1.0 in stage 672.0 (TID 5377). 2260 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 1.0 in stage 672.0 (TID 5377) in 4 ms on localhost (7/8)
16/09/07 19:10:41 INFO Executor: Finished task 0.0 in stage 672.0 (TID 5376). 2260 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 0.0 in stage 672.0 (TID 5376) in 4 ms on localhost (8/8)
16/09/07 19:10:41 INFO DAGScheduler: ShuffleMapStage 672 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:41 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:41 INFO DAGScheduler: running: Set()
16/09/07 19:10:41 INFO TaskSchedulerImpl: Removed TaskSet 672.0, whose tasks have all completed, from pool 
16/09/07 19:10:41 INFO DAGScheduler: waiting: Set(ResultStage 673)
16/09/07 19:10:41 INFO DAGScheduler: failed: Set()
16/09/07 19:10:41 INFO DAGScheduler: Missing parents for ResultStage 673: List()
16/09/07 19:10:41 INFO DAGScheduler: Submitting ResultStage 673 (MapPartitionsRDD[2067] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:41 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19037520, maxMem=4042889625
16/09/07 19:10:41 INFO MemoryStore: Block broadcast_1018 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:41 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=19042616, maxMem=4042889625
16/09/07 19:10:41 INFO MemoryStore: Block broadcast_1018_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:41 INFO BlockManagerInfo: Added broadcast_1018_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:41 INFO SparkContext: Created broadcast 1018 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:41 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 673 (MapPartitionsRDD[2067] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:41 INFO TaskSchedulerImpl: Adding task set 673.0 with 8 tasks
16/09/07 19:10:41 INFO TaskSetManager: Starting task 0.0 in stage 673.0 (TID 5384, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 1.0 in stage 673.0 (TID 5385, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 2.0 in stage 673.0 (TID 5386, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 3.0 in stage 673.0 (TID 5387, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 4.0 in stage 673.0 (TID 5388, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 5.0 in stage 673.0 (TID 5389, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 6.0 in stage 673.0 (TID 5390, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:41 INFO TaskSetManager: Starting task 7.0 in stage 673.0 (TID 5391, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:41 INFO Executor: Running task 1.0 in stage 673.0 (TID 5385)
16/09/07 19:10:41 INFO Executor: Running task 5.0 in stage 673.0 (TID 5389)
16/09/07 19:10:41 INFO Executor: Running task 7.0 in stage 673.0 (TID 5391)
16/09/07 19:10:41 INFO Executor: Running task 6.0 in stage 673.0 (TID 5390)
16/09/07 19:10:41 INFO Executor: Running task 0.0 in stage 673.0 (TID 5384)
16/09/07 19:10:41 INFO Executor: Running task 2.0 in stage 673.0 (TID 5386)
16/09/07 19:10:41 INFO Executor: Running task 4.0 in stage 673.0 (TID 5388)
16/09/07 19:10:41 INFO Executor: Running task 3.0 in stage 673.0 (TID 5387)
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:41 INFO Executor: Finished task 4.0 in stage 673.0 (TID 5388). 2160 bytes result sent to driver
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:41 INFO TaskSetManager: Finished task 4.0 in stage 673.0 (TID 5388) in 1 ms on localhost (1/8)
16/09/07 19:10:41 INFO Executor: Finished task 3.0 in stage 673.0 (TID 5387). 2160 bytes result sent to driver
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:41 INFO Executor: Finished task 1.0 in stage 673.0 (TID 5385). 2160 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 3.0 in stage 673.0 (TID 5387) in 2 ms on localhost (2/8)
16/09/07 19:10:41 INFO TaskSetManager: Finished task 1.0 in stage 673.0 (TID 5385) in 3 ms on localhost (3/8)
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:41 INFO Executor: Finished task 2.0 in stage 673.0 (TID 5386). 2160 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 2.0 in stage 673.0 (TID 5386) in 3 ms on localhost (4/8)
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:41 INFO Executor: Finished task 5.0 in stage 673.0 (TID 5389). 2160 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 5.0 in stage 673.0 (TID 5389) in 3 ms on localhost (5/8)
16/09/07 19:10:41 INFO BlockManagerInfo: Removed broadcast_1017_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:41 INFO Executor: Finished task 0.0 in stage 673.0 (TID 5384). 3485274 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 0.0 in stage 673.0 (TID 5384) in 824 ms on localhost (6/8)
16/09/07 19:10:41 INFO Executor: Finished task 7.0 in stage 673.0 (TID 5391). 3485274 bytes result sent to driver
16/09/07 19:10:41 INFO Executor: Finished task 6.0 in stage 673.0 (TID 5390). 3485274 bytes result sent to driver
16/09/07 19:10:41 INFO TaskSetManager: Finished task 7.0 in stage 673.0 (TID 5391) in 868 ms on localhost (7/8)
16/09/07 19:10:41 INFO TaskSetManager: Finished task 6.0 in stage 673.0 (TID 5390) in 872 ms on localhost (8/8)
16/09/07 19:10:41 INFO TaskSchedulerImpl: Removed TaskSet 673.0, whose tasks have all completed, from pool 
16/09/07 19:10:41 INFO DAGScheduler: ResultStage 673 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.873 s
16/09/07 19:10:41 INFO DAGScheduler: Job 341 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.880612 s
16/09/07 19:10:41 INFO ParameterAveragingTrainingMaster: Completed training of split 20 of 78
16/09/07 19:10:41 INFO ParameterAveragingTrainingMaster: Starting training of split 21 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:42 INFO MemoryStore: ensureFreeSpace(18220176) called with curMem=19038332, maxMem=4042889625
16/09/07 19:10:42 INFO MemoryStore: Block broadcast_1019 stored as values in memory (estimated size 17.4 MB, free 3.7 GB)
16/09/07 19:10:42 INFO MemoryStore: ensureFreeSpace(276107) called with curMem=37258508, maxMem=4042889625
16/09/07 19:10:42 INFO MemoryStore: Block broadcast_1019_piece0 stored as bytes in memory (estimated size 269.6 KB, free 3.7 GB)
16/09/07 19:10:42 INFO BlockManagerInfo: Added broadcast_1019_piece0 in memory on localhost:41251 (size: 269.6 KB, free: 3.8 GB)
16/09/07 19:10:42 INFO SparkContext: Created broadcast 1019 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:42 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:42 INFO DAGScheduler: Registering RDD 2068 (repartition at SparkUtils.java:190)
16/09/07 19:10:42 INFO DAGScheduler: Got job 342 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:42 INFO DAGScheduler: Final stage: ResultStage 675(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:42 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 674)
16/09/07 19:10:42 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 674)
16/09/07 19:10:42 INFO DAGScheduler: Submitting ShuffleMapStage 674 (MapPartitionsRDD[2068] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:42 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=37534615, maxMem=4042889625
16/09/07 19:10:42 INFO MemoryStore: Block broadcast_1020 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:42 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=37539079, maxMem=4042889625
16/09/07 19:10:42 INFO MemoryStore: Block broadcast_1020_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:42 INFO BlockManagerInfo: Added broadcast_1020_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:42 INFO SparkContext: Created broadcast 1020 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:42 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 674 (MapPartitionsRDD[2068] at repartition at SparkUtils.java:190)
16/09/07 19:10:42 INFO TaskSchedulerImpl: Adding task set 674.0 with 8 tasks
16/09/07 19:10:42 INFO TaskSetManager: Starting task 0.0 in stage 674.0 (TID 5392, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 1.0 in stage 674.0 (TID 5393, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 2.0 in stage 674.0 (TID 5394, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 3.0 in stage 674.0 (TID 5395, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 4.0 in stage 674.0 (TID 5396, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 5.0 in stage 674.0 (TID 5397, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 6.0 in stage 674.0 (TID 5398, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 7.0 in stage 674.0 (TID 5399, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:42 INFO Executor: Running task 3.0 in stage 674.0 (TID 5395)
16/09/07 19:10:42 INFO Executor: Running task 5.0 in stage 674.0 (TID 5397)
16/09/07 19:10:42 INFO Executor: Running task 7.0 in stage 674.0 (TID 5399)
16/09/07 19:10:42 INFO Executor: Running task 2.0 in stage 674.0 (TID 5394)
16/09/07 19:10:42 INFO Executor: Running task 4.0 in stage 674.0 (TID 5396)
16/09/07 19:10:42 INFO Executor: Running task 1.0 in stage 674.0 (TID 5393)
16/09/07 19:10:42 INFO Executor: Running task 0.0 in stage 674.0 (TID 5392)
16/09/07 19:10:42 INFO Executor: Running task 6.0 in stage 674.0 (TID 5398)
16/09/07 19:10:42 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:42 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:42 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:42 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:42 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:42 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:42 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:42 INFO Executor: Finished task 1.0 in stage 674.0 (TID 5393). 2260 bytes result sent to driver
16/09/07 19:10:42 INFO Executor: Finished task 4.0 in stage 674.0 (TID 5396). 2260 bytes result sent to driver
16/09/07 19:10:42 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:42 INFO Executor: Finished task 7.0 in stage 674.0 (TID 5399). 2260 bytes result sent to driver
16/09/07 19:10:42 INFO TaskSetManager: Finished task 1.0 in stage 674.0 (TID 5393) in 3 ms on localhost (1/8)
16/09/07 19:10:42 INFO TaskSetManager: Finished task 4.0 in stage 674.0 (TID 5396) in 3 ms on localhost (2/8)
16/09/07 19:10:42 INFO TaskSetManager: Finished task 7.0 in stage 674.0 (TID 5399) in 3 ms on localhost (3/8)
16/09/07 19:10:42 INFO Executor: Finished task 0.0 in stage 674.0 (TID 5392). 2260 bytes result sent to driver
16/09/07 19:10:42 INFO TaskSetManager: Finished task 0.0 in stage 674.0 (TID 5392) in 3 ms on localhost (4/8)
16/09/07 19:10:42 INFO Executor: Finished task 2.0 in stage 674.0 (TID 5394). 2260 bytes result sent to driver
16/09/07 19:10:42 INFO TaskSetManager: Finished task 2.0 in stage 674.0 (TID 5394) in 3 ms on localhost (5/8)
16/09/07 19:10:42 INFO Executor: Finished task 6.0 in stage 674.0 (TID 5398). 2260 bytes result sent to driver
16/09/07 19:10:42 INFO TaskSetManager: Finished task 6.0 in stage 674.0 (TID 5398) in 4 ms on localhost (6/8)
16/09/07 19:10:42 INFO Executor: Finished task 3.0 in stage 674.0 (TID 5395). 2260 bytes result sent to driver
16/09/07 19:10:42 INFO Executor: Finished task 5.0 in stage 674.0 (TID 5397). 2260 bytes result sent to driver
16/09/07 19:10:42 INFO TaskSetManager: Finished task 5.0 in stage 674.0 (TID 5397) in 4 ms on localhost (7/8)
16/09/07 19:10:42 INFO TaskSetManager: Finished task 3.0 in stage 674.0 (TID 5395) in 4 ms on localhost (8/8)
16/09/07 19:10:42 INFO DAGScheduler: ShuffleMapStage 674 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:42 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:42 INFO TaskSchedulerImpl: Removed TaskSet 674.0, whose tasks have all completed, from pool 
16/09/07 19:10:42 INFO DAGScheduler: running: Set()
16/09/07 19:10:42 INFO DAGScheduler: waiting: Set(ResultStage 675)
16/09/07 19:10:42 INFO DAGScheduler: failed: Set()
16/09/07 19:10:42 INFO DAGScheduler: Missing parents for ResultStage 675: List()
16/09/07 19:10:42 INFO DAGScheduler: Submitting ResultStage 675 (MapPartitionsRDD[2072] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:42 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=37541642, maxMem=4042889625
16/09/07 19:10:42 INFO MemoryStore: Block broadcast_1021 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:42 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=37546738, maxMem=4042889625
16/09/07 19:10:42 INFO MemoryStore: Block broadcast_1021_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:42 INFO BlockManagerInfo: Added broadcast_1021_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:42 INFO SparkContext: Created broadcast 1021 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:42 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 675 (MapPartitionsRDD[2072] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:42 INFO TaskSchedulerImpl: Adding task set 675.0 with 8 tasks
16/09/07 19:10:42 INFO TaskSetManager: Starting task 0.0 in stage 675.0 (TID 5400, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 1.0 in stage 675.0 (TID 5401, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 2.0 in stage 675.0 (TID 5402, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 3.0 in stage 675.0 (TID 5403, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 4.0 in stage 675.0 (TID 5404, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 5.0 in stage 675.0 (TID 5405, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 6.0 in stage 675.0 (TID 5406, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:42 INFO TaskSetManager: Starting task 7.0 in stage 675.0 (TID 5407, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:42 INFO Executor: Running task 0.0 in stage 675.0 (TID 5400)
16/09/07 19:10:42 INFO Executor: Running task 3.0 in stage 675.0 (TID 5403)
16/09/07 19:10:42 INFO Executor: Running task 2.0 in stage 675.0 (TID 5402)
16/09/07 19:10:42 INFO Executor: Running task 5.0 in stage 675.0 (TID 5405)
16/09/07 19:10:42 INFO Executor: Running task 1.0 in stage 675.0 (TID 5401)
16/09/07 19:10:42 INFO Executor: Running task 6.0 in stage 675.0 (TID 5406)
16/09/07 19:10:42 INFO Executor: Running task 4.0 in stage 675.0 (TID 5404)
16/09/07 19:10:42 INFO Executor: Running task 7.0 in stage 675.0 (TID 5407)
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:42 INFO Executor: Finished task 1.0 in stage 675.0 (TID 5401). 2160 bytes result sent to driver
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:42 INFO TaskSetManager: Finished task 1.0 in stage 675.0 (TID 5401) in 2 ms on localhost (1/8)
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:42 INFO Executor: Finished task 4.0 in stage 675.0 (TID 5404). 2160 bytes result sent to driver
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:42 INFO TaskSetManager: Finished task 4.0 in stage 675.0 (TID 5404) in 2 ms on localhost (2/8)
16/09/07 19:10:42 INFO Executor: Finished task 3.0 in stage 675.0 (TID 5403). 2160 bytes result sent to driver
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:42 INFO TaskSetManager: Finished task 3.0 in stage 675.0 (TID 5403) in 2 ms on localhost (3/8)
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:42 INFO Executor: Finished task 5.0 in stage 675.0 (TID 5405). 2160 bytes result sent to driver
16/09/07 19:10:42 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:42 INFO Executor: Finished task 2.0 in stage 675.0 (TID 5402). 2160 bytes result sent to driver
16/09/07 19:10:42 INFO TaskSetManager: Finished task 5.0 in stage 675.0 (TID 5405) in 2 ms on localhost (4/8)
16/09/07 19:10:42 INFO TaskSetManager: Finished task 2.0 in stage 675.0 (TID 5402) in 3 ms on localhost (5/8)
16/09/07 19:10:42 INFO BlockManagerInfo: Removed broadcast_1020_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:42 INFO BlockManagerInfo: Removed broadcast_1018_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:42 INFO ContextCleaner: Cleaned accumulator 674
16/09/07 19:10:42 INFO ContextCleaner: Cleaned accumulator 673
16/09/07 19:10:42 INFO BlockManagerInfo: Removed broadcast_1016_piece0 on localhost:41251 in memory (size: 269.9 KB, free: 3.8 GB)
16/09/07 19:10:42 INFO ContextCleaner: Cleaned shuffle 331
16/09/07 19:10:43 INFO Executor: Finished task 7.0 in stage 675.0 (TID 5407). 3485524 bytes result sent to driver
16/09/07 19:10:43 INFO TaskSetManager: Finished task 7.0 in stage 675.0 (TID 5407) in 891 ms on localhost (6/8)
16/09/07 19:10:43 INFO Executor: Finished task 6.0 in stage 675.0 (TID 5406). 3485524 bytes result sent to driver
16/09/07 19:10:43 INFO Executor: Finished task 0.0 in stage 675.0 (TID 5400). 3485524 bytes result sent to driver
16/09/07 19:10:43 INFO TaskSetManager: Finished task 6.0 in stage 675.0 (TID 5406) in 945 ms on localhost (7/8)
16/09/07 19:10:43 INFO TaskSetManager: Finished task 0.0 in stage 675.0 (TID 5400) in 946 ms on localhost (8/8)
16/09/07 19:10:43 INFO TaskSchedulerImpl: Removed TaskSet 675.0, whose tasks have all completed, from pool 
16/09/07 19:10:43 INFO DAGScheduler: ResultStage 675 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.946 s
16/09/07 19:10:43 INFO DAGScheduler: Job 342 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.954320 s
16/09/07 19:10:43 INFO ParameterAveragingTrainingMaster: Completed training of split 21 of 78
16/09/07 19:10:43 INFO ParameterAveragingTrainingMaster: Starting training of split 22 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:43 INFO MemoryStore: ensureFreeSpace(18243968) called with curMem=19047946, maxMem=4042889625
16/09/07 19:10:43 INFO MemoryStore: Block broadcast_1022 stored as values in memory (estimated size 17.4 MB, free 3.7 GB)
16/09/07 19:10:43 INFO BlockManagerInfo: Removed broadcast_1021_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:43 INFO ContextCleaner: Cleaned accumulator 676
16/09/07 19:10:43 INFO ContextCleaner: Cleaned accumulator 675
16/09/07 19:10:43 INFO BlockManagerInfo: Removed broadcast_1019_piece0 on localhost:41251 in memory (size: 269.6 KB, free: 3.8 GB)
16/09/07 19:10:43 INFO ContextCleaner: Cleaned shuffle 332
16/09/07 19:10:43 INFO MemoryStore: ensureFreeSpace(276485) called with curMem=18787792, maxMem=4042889625
16/09/07 19:10:43 INFO MemoryStore: Block broadcast_1022_piece0 stored as bytes in memory (estimated size 270.0 KB, free 3.7 GB)
16/09/07 19:10:43 INFO BlockManagerInfo: Added broadcast_1022_piece0 in memory on localhost:41251 (size: 270.0 KB, free: 3.8 GB)
16/09/07 19:10:43 INFO SparkContext: Created broadcast 1022 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:43 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:43 INFO DAGScheduler: Registering RDD 2073 (repartition at SparkUtils.java:190)
16/09/07 19:10:43 INFO DAGScheduler: Got job 343 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:43 INFO DAGScheduler: Final stage: ResultStage 677(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:43 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 676)
16/09/07 19:10:43 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 676)
16/09/07 19:10:43 INFO DAGScheduler: Submitting ShuffleMapStage 676 (MapPartitionsRDD[2073] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:43 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19064277, maxMem=4042889625
16/09/07 19:10:43 INFO MemoryStore: Block broadcast_1023 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:43 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19068741, maxMem=4042889625
16/09/07 19:10:43 INFO MemoryStore: Block broadcast_1023_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:43 INFO BlockManagerInfo: Added broadcast_1023_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:43 INFO SparkContext: Created broadcast 1023 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:43 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 676 (MapPartitionsRDD[2073] at repartition at SparkUtils.java:190)
16/09/07 19:10:43 INFO TaskSchedulerImpl: Adding task set 676.0 with 8 tasks
16/09/07 19:10:43 INFO TaskSetManager: Starting task 0.0 in stage 676.0 (TID 5408, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 1.0 in stage 676.0 (TID 5409, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 2.0 in stage 676.0 (TID 5410, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 3.0 in stage 676.0 (TID 5411, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 4.0 in stage 676.0 (TID 5412, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 5.0 in stage 676.0 (TID 5413, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 6.0 in stage 676.0 (TID 5414, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 7.0 in stage 676.0 (TID 5415, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:43 INFO Executor: Running task 0.0 in stage 676.0 (TID 5408)
16/09/07 19:10:43 INFO Executor: Running task 2.0 in stage 676.0 (TID 5410)
16/09/07 19:10:43 INFO Executor: Running task 3.0 in stage 676.0 (TID 5411)
16/09/07 19:10:43 INFO Executor: Running task 4.0 in stage 676.0 (TID 5412)
16/09/07 19:10:43 INFO Executor: Running task 5.0 in stage 676.0 (TID 5413)
16/09/07 19:10:43 INFO Executor: Running task 1.0 in stage 676.0 (TID 5409)
16/09/07 19:10:43 INFO Executor: Running task 6.0 in stage 676.0 (TID 5414)
16/09/07 19:10:43 INFO Executor: Running task 7.0 in stage 676.0 (TID 5415)
16/09/07 19:10:43 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:43 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:43 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:43 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:43 INFO Executor: Finished task 3.0 in stage 676.0 (TID 5411). 2260 bytes result sent to driver
16/09/07 19:10:43 INFO Executor: Finished task 0.0 in stage 676.0 (TID 5408). 2260 bytes result sent to driver
16/09/07 19:10:43 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:43 INFO TaskSetManager: Finished task 3.0 in stage 676.0 (TID 5411) in 3 ms on localhost (1/8)
16/09/07 19:10:43 INFO TaskSetManager: Finished task 0.0 in stage 676.0 (TID 5408) in 3 ms on localhost (2/8)
16/09/07 19:10:43 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:43 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:43 INFO Executor: Finished task 6.0 in stage 676.0 (TID 5414). 2260 bytes result sent to driver
16/09/07 19:10:43 INFO TaskSetManager: Finished task 6.0 in stage 676.0 (TID 5414) in 3 ms on localhost (3/8)
16/09/07 19:10:43 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:43 INFO Executor: Finished task 5.0 in stage 676.0 (TID 5413). 2260 bytes result sent to driver
16/09/07 19:10:43 INFO Executor: Finished task 4.0 in stage 676.0 (TID 5412). 2260 bytes result sent to driver
16/09/07 19:10:43 INFO Executor: Finished task 2.0 in stage 676.0 (TID 5410). 2260 bytes result sent to driver
16/09/07 19:10:43 INFO Executor: Finished task 7.0 in stage 676.0 (TID 5415). 2260 bytes result sent to driver
16/09/07 19:10:43 INFO TaskSetManager: Finished task 5.0 in stage 676.0 (TID 5413) in 5 ms on localhost (4/8)
16/09/07 19:10:43 INFO TaskSetManager: Finished task 4.0 in stage 676.0 (TID 5412) in 5 ms on localhost (5/8)
16/09/07 19:10:43 INFO TaskSetManager: Finished task 2.0 in stage 676.0 (TID 5410) in 5 ms on localhost (6/8)
16/09/07 19:10:43 INFO TaskSetManager: Finished task 7.0 in stage 676.0 (TID 5415) in 4 ms on localhost (7/8)
16/09/07 19:10:43 INFO Executor: Finished task 1.0 in stage 676.0 (TID 5409). 2260 bytes result sent to driver
16/09/07 19:10:43 INFO TaskSetManager: Finished task 1.0 in stage 676.0 (TID 5409) in 5 ms on localhost (8/8)
16/09/07 19:10:43 INFO TaskSchedulerImpl: Removed TaskSet 676.0, whose tasks have all completed, from pool 
16/09/07 19:10:43 INFO DAGScheduler: ShuffleMapStage 676 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:43 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:43 INFO DAGScheduler: running: Set()
16/09/07 19:10:43 INFO DAGScheduler: waiting: Set(ResultStage 677)
16/09/07 19:10:43 INFO DAGScheduler: failed: Set()
16/09/07 19:10:43 INFO DAGScheduler: Missing parents for ResultStage 677: List()
16/09/07 19:10:43 INFO DAGScheduler: Submitting ResultStage 677 (MapPartitionsRDD[2077] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:43 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19071305, maxMem=4042889625
16/09/07 19:10:43 INFO MemoryStore: Block broadcast_1024 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:43 INFO MemoryStore: ensureFreeSpace(2746) called with curMem=19076401, maxMem=4042889625
16/09/07 19:10:43 INFO MemoryStore: Block broadcast_1024_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:43 INFO BlockManagerInfo: Added broadcast_1024_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:43 INFO SparkContext: Created broadcast 1024 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:43 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 677 (MapPartitionsRDD[2077] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:43 INFO TaskSchedulerImpl: Adding task set 677.0 with 8 tasks
16/09/07 19:10:43 INFO TaskSetManager: Starting task 0.0 in stage 677.0 (TID 5416, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 1.0 in stage 677.0 (TID 5417, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 2.0 in stage 677.0 (TID 5418, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 3.0 in stage 677.0 (TID 5419, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 4.0 in stage 677.0 (TID 5420, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 5.0 in stage 677.0 (TID 5421, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 6.0 in stage 677.0 (TID 5422, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:43 INFO TaskSetManager: Starting task 7.0 in stage 677.0 (TID 5423, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:43 INFO Executor: Running task 2.0 in stage 677.0 (TID 5418)
16/09/07 19:10:43 INFO Executor: Running task 0.0 in stage 677.0 (TID 5416)
16/09/07 19:10:43 INFO Executor: Running task 5.0 in stage 677.0 (TID 5421)
16/09/07 19:10:43 INFO Executor: Running task 1.0 in stage 677.0 (TID 5417)
16/09/07 19:10:43 INFO Executor: Running task 4.0 in stage 677.0 (TID 5420)
16/09/07 19:10:43 INFO Executor: Running task 3.0 in stage 677.0 (TID 5419)
16/09/07 19:10:43 INFO Executor: Running task 6.0 in stage 677.0 (TID 5422)
16/09/07 19:10:43 INFO Executor: Running task 7.0 in stage 677.0 (TID 5423)
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:43 INFO Executor: Finished task 1.0 in stage 677.0 (TID 5417). 2160 bytes result sent to driver
16/09/07 19:10:43 INFO Executor: Finished task 3.0 in stage 677.0 (TID 5419). 2160 bytes result sent to driver
16/09/07 19:10:43 INFO Executor: Finished task 4.0 in stage 677.0 (TID 5420). 2160 bytes result sent to driver
16/09/07 19:10:43 INFO TaskSetManager: Finished task 1.0 in stage 677.0 (TID 5417) in 2 ms on localhost (1/8)
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:43 INFO TaskSetManager: Finished task 3.0 in stage 677.0 (TID 5419) in 2 ms on localhost (2/8)
16/09/07 19:10:43 INFO TaskSetManager: Finished task 4.0 in stage 677.0 (TID 5420) in 2 ms on localhost (3/8)
16/09/07 19:10:43 INFO Executor: Finished task 5.0 in stage 677.0 (TID 5421). 2160 bytes result sent to driver
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:43 INFO TaskSetManager: Finished task 5.0 in stage 677.0 (TID 5421) in 2 ms on localhost (4/8)
16/09/07 19:10:43 INFO Executor: Finished task 2.0 in stage 677.0 (TID 5418). 2160 bytes result sent to driver
16/09/07 19:10:43 INFO TaskSetManager: Finished task 2.0 in stage 677.0 (TID 5418) in 3 ms on localhost (5/8)
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:43 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:43 INFO BlockManagerInfo: Removed broadcast_1023_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:44 INFO Executor: Finished task 6.0 in stage 677.0 (TID 5422). 3485274 bytes result sent to driver
16/09/07 19:10:44 INFO TaskSetManager: Finished task 6.0 in stage 677.0 (TID 5422) in 885 ms on localhost (6/8)
16/09/07 19:10:44 INFO Executor: Finished task 0.0 in stage 677.0 (TID 5416). 3485274 bytes result sent to driver
16/09/07 19:10:44 INFO Executor: Finished task 7.0 in stage 677.0 (TID 5423). 3485274 bytes result sent to driver
16/09/07 19:10:44 INFO TaskSetManager: Finished task 0.0 in stage 677.0 (TID 5416) in 928 ms on localhost (7/8)
16/09/07 19:10:44 INFO TaskSetManager: Finished task 7.0 in stage 677.0 (TID 5423) in 939 ms on localhost (8/8)
16/09/07 19:10:44 INFO TaskSchedulerImpl: Removed TaskSet 677.0, whose tasks have all completed, from pool 
16/09/07 19:10:44 INFO DAGScheduler: ResultStage 677 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.939 s
16/09/07 19:10:44 INFO DAGScheduler: Job 343 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.948507 s
16/09/07 19:10:44 INFO ParameterAveragingTrainingMaster: Completed training of split 22 of 78
16/09/07 19:10:44 INFO ParameterAveragingTrainingMaster: Starting training of split 23 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:44 INFO BlockManagerInfo: Removed broadcast_1024_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:44 INFO ContextCleaner: Cleaned accumulator 678
16/09/07 19:10:44 INFO ContextCleaner: Cleaned accumulator 677
16/09/07 19:10:44 INFO BlockManagerInfo: Removed broadcast_1022_piece0 on localhost:41251 in memory (size: 270.0 KB, free: 3.8 GB)
16/09/07 19:10:44 INFO ContextCleaner: Cleaned shuffle 333
16/09/07 19:10:44 INFO MemoryStore: ensureFreeSpace(17970968) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:44 INFO MemoryStore: Block broadcast_1025 stored as values in memory (estimated size 17.1 MB, free 3.7 GB)
16/09/07 19:10:44 INFO MemoryStore: ensureFreeSpace(261228) called with curMem=18514792, maxMem=4042889625
16/09/07 19:10:44 INFO MemoryStore: Block broadcast_1025_piece0 stored as bytes in memory (estimated size 255.1 KB, free 3.7 GB)
16/09/07 19:10:44 INFO BlockManagerInfo: Added broadcast_1025_piece0 in memory on localhost:41251 (size: 255.1 KB, free: 3.8 GB)
16/09/07 19:10:44 INFO SparkContext: Created broadcast 1025 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:44 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:44 INFO DAGScheduler: Registering RDD 2078 (repartition at SparkUtils.java:190)
16/09/07 19:10:44 INFO DAGScheduler: Got job 344 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:44 INFO DAGScheduler: Final stage: ResultStage 679(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:44 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 678)
16/09/07 19:10:44 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 678)
16/09/07 19:10:44 INFO DAGScheduler: Submitting ShuffleMapStage 678 (MapPartitionsRDD[2078] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:44 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18776020, maxMem=4042889625
16/09/07 19:10:44 INFO MemoryStore: Block broadcast_1026 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:44 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=18780484, maxMem=4042889625
16/09/07 19:10:44 INFO MemoryStore: Block broadcast_1026_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:44 INFO BlockManagerInfo: Added broadcast_1026_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:44 INFO SparkContext: Created broadcast 1026 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:44 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 678 (MapPartitionsRDD[2078] at repartition at SparkUtils.java:190)
16/09/07 19:10:44 INFO TaskSchedulerImpl: Adding task set 678.0 with 8 tasks
16/09/07 19:10:44 INFO TaskSetManager: Starting task 0.0 in stage 678.0 (TID 5424, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 1.0 in stage 678.0 (TID 5425, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 2.0 in stage 678.0 (TID 5426, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 3.0 in stage 678.0 (TID 5427, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 4.0 in stage 678.0 (TID 5428, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 5.0 in stage 678.0 (TID 5429, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 6.0 in stage 678.0 (TID 5430, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 7.0 in stage 678.0 (TID 5431, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:44 INFO Executor: Running task 0.0 in stage 678.0 (TID 5424)
16/09/07 19:10:44 INFO Executor: Running task 2.0 in stage 678.0 (TID 5426)
16/09/07 19:10:44 INFO Executor: Running task 3.0 in stage 678.0 (TID 5427)
16/09/07 19:10:44 INFO Executor: Running task 4.0 in stage 678.0 (TID 5428)
16/09/07 19:10:44 INFO Executor: Running task 5.0 in stage 678.0 (TID 5429)
16/09/07 19:10:44 INFO Executor: Running task 1.0 in stage 678.0 (TID 5425)
16/09/07 19:10:44 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:44 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:44 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:44 INFO Executor: Finished task 5.0 in stage 678.0 (TID 5429). 2260 bytes result sent to driver
16/09/07 19:10:44 INFO Executor: Running task 6.0 in stage 678.0 (TID 5430)
16/09/07 19:10:44 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:44 INFO TaskSetManager: Finished task 5.0 in stage 678.0 (TID 5429) in 2 ms on localhost (1/8)
16/09/07 19:10:44 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:44 INFO Executor: Finished task 0.0 in stage 678.0 (TID 5424). 2260 bytes result sent to driver
16/09/07 19:10:44 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:44 INFO TaskSetManager: Finished task 0.0 in stage 678.0 (TID 5424) in 3 ms on localhost (2/8)
16/09/07 19:10:44 INFO Executor: Finished task 1.0 in stage 678.0 (TID 5425). 2260 bytes result sent to driver
16/09/07 19:10:44 INFO Executor: Running task 7.0 in stage 678.0 (TID 5431)
16/09/07 19:10:44 INFO Executor: Finished task 2.0 in stage 678.0 (TID 5426). 2260 bytes result sent to driver
16/09/07 19:10:44 INFO Executor: Finished task 3.0 in stage 678.0 (TID 5427). 2260 bytes result sent to driver
16/09/07 19:10:44 INFO TaskSetManager: Finished task 1.0 in stage 678.0 (TID 5425) in 3 ms on localhost (3/8)
16/09/07 19:10:44 INFO TaskSetManager: Finished task 2.0 in stage 678.0 (TID 5426) in 3 ms on localhost (4/8)
16/09/07 19:10:44 INFO TaskSetManager: Finished task 3.0 in stage 678.0 (TID 5427) in 3 ms on localhost (5/8)
16/09/07 19:10:44 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:44 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:44 INFO Executor: Finished task 4.0 in stage 678.0 (TID 5428). 2260 bytes result sent to driver
16/09/07 19:10:44 INFO Executor: Finished task 6.0 in stage 678.0 (TID 5430). 2260 bytes result sent to driver
16/09/07 19:10:44 INFO TaskSetManager: Finished task 4.0 in stage 678.0 (TID 5428) in 4 ms on localhost (6/8)
16/09/07 19:10:44 INFO Executor: Finished task 7.0 in stage 678.0 (TID 5431). 2260 bytes result sent to driver
16/09/07 19:10:44 INFO TaskSetManager: Finished task 6.0 in stage 678.0 (TID 5430) in 4 ms on localhost (7/8)
16/09/07 19:10:44 INFO TaskSetManager: Finished task 7.0 in stage 678.0 (TID 5431) in 4 ms on localhost (8/8)
16/09/07 19:10:44 INFO TaskSchedulerImpl: Removed TaskSet 678.0, whose tasks have all completed, from pool 
16/09/07 19:10:44 INFO DAGScheduler: ShuffleMapStage 678 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:44 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:44 INFO DAGScheduler: running: Set()
16/09/07 19:10:44 INFO DAGScheduler: waiting: Set(ResultStage 679)
16/09/07 19:10:44 INFO DAGScheduler: failed: Set()
16/09/07 19:10:44 INFO DAGScheduler: Missing parents for ResultStage 679: List()
16/09/07 19:10:44 INFO DAGScheduler: Submitting ResultStage 679 (MapPartitionsRDD[2082] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:44 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18783047, maxMem=4042889625
16/09/07 19:10:44 INFO MemoryStore: Block broadcast_1027 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:44 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18788143, maxMem=4042889625
16/09/07 19:10:44 INFO MemoryStore: Block broadcast_1027_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:44 INFO BlockManagerInfo: Added broadcast_1027_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:44 INFO SparkContext: Created broadcast 1027 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:44 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 679 (MapPartitionsRDD[2082] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:44 INFO TaskSchedulerImpl: Adding task set 679.0 with 8 tasks
16/09/07 19:10:44 INFO TaskSetManager: Starting task 0.0 in stage 679.0 (TID 5432, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 1.0 in stage 679.0 (TID 5433, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 2.0 in stage 679.0 (TID 5434, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 3.0 in stage 679.0 (TID 5435, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 4.0 in stage 679.0 (TID 5436, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 5.0 in stage 679.0 (TID 5437, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 6.0 in stage 679.0 (TID 5438, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:44 INFO TaskSetManager: Starting task 7.0 in stage 679.0 (TID 5439, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:44 INFO Executor: Running task 0.0 in stage 679.0 (TID 5432)
16/09/07 19:10:44 INFO Executor: Running task 5.0 in stage 679.0 (TID 5437)
16/09/07 19:10:44 INFO Executor: Running task 7.0 in stage 679.0 (TID 5439)
16/09/07 19:10:44 INFO Executor: Running task 1.0 in stage 679.0 (TID 5433)
16/09/07 19:10:44 INFO Executor: Running task 3.0 in stage 679.0 (TID 5435)
16/09/07 19:10:44 INFO Executor: Running task 2.0 in stage 679.0 (TID 5434)
16/09/07 19:10:44 INFO Executor: Running task 4.0 in stage 679.0 (TID 5436)
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:44 INFO Executor: Finished task 4.0 in stage 679.0 (TID 5436). 2160 bytes result sent to driver
16/09/07 19:10:44 INFO Executor: Running task 6.0 in stage 679.0 (TID 5438)
16/09/07 19:10:44 INFO Executor: Finished task 2.0 in stage 679.0 (TID 5434). 2160 bytes result sent to driver
16/09/07 19:10:44 INFO Executor: Finished task 5.0 in stage 679.0 (TID 5437). 2160 bytes result sent to driver
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:44 INFO TaskSetManager: Finished task 4.0 in stage 679.0 (TID 5436) in 3 ms on localhost (1/8)
16/09/07 19:10:44 INFO Executor: Finished task 1.0 in stage 679.0 (TID 5433). 2160 bytes result sent to driver
16/09/07 19:10:44 INFO TaskSetManager: Finished task 2.0 in stage 679.0 (TID 5434) in 4 ms on localhost (2/8)
16/09/07 19:10:44 INFO Executor: Finished task 3.0 in stage 679.0 (TID 5435). 2160 bytes result sent to driver
16/09/07 19:10:44 INFO TaskSetManager: Finished task 5.0 in stage 679.0 (TID 5437) in 3 ms on localhost (3/8)
16/09/07 19:10:44 INFO TaskSetManager: Finished task 1.0 in stage 679.0 (TID 5433) in 4 ms on localhost (4/8)
16/09/07 19:10:44 INFO TaskSetManager: Finished task 3.0 in stage 679.0 (TID 5435) in 4 ms on localhost (5/8)
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:44 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:44 INFO BlockManagerInfo: Removed broadcast_1026_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:45 INFO Executor: Finished task 7.0 in stage 679.0 (TID 5439). 3485274 bytes result sent to driver
16/09/07 19:10:45 INFO TaskSetManager: Finished task 7.0 in stage 679.0 (TID 5439) in 878 ms on localhost (6/8)
16/09/07 19:10:45 INFO Executor: Finished task 0.0 in stage 679.0 (TID 5432). 3485274 bytes result sent to driver
16/09/07 19:10:45 INFO Executor: Finished task 6.0 in stage 679.0 (TID 5438). 3485274 bytes result sent to driver
16/09/07 19:10:45 INFO TaskSetManager: Finished task 0.0 in stage 679.0 (TID 5432) in 950 ms on localhost (7/8)
16/09/07 19:10:45 INFO TaskSetManager: Finished task 6.0 in stage 679.0 (TID 5438) in 949 ms on localhost (8/8)
16/09/07 19:10:45 INFO TaskSchedulerImpl: Removed TaskSet 679.0, whose tasks have all completed, from pool 
16/09/07 19:10:45 INFO DAGScheduler: ResultStage 679 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.950 s
16/09/07 19:10:45 INFO DAGScheduler: Job 344 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.959084 s
16/09/07 19:10:45 INFO ParameterAveragingTrainingMaster: Completed training of split 23 of 78
16/09/07 19:10:45 INFO ParameterAveragingTrainingMaster: Starting training of split 24 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:45 INFO BlockManagerInfo: Removed broadcast_1027_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:45 INFO ContextCleaner: Cleaned accumulator 680
16/09/07 19:10:45 INFO ContextCleaner: Cleaned accumulator 679
16/09/07 19:10:45 INFO BlockManagerInfo: Removed broadcast_1025_piece0 on localhost:41251 in memory (size: 255.1 KB, free: 3.8 GB)
16/09/07 19:10:45 INFO ContextCleaner: Cleaned shuffle 334
16/09/07 19:10:45 INFO MemoryStore: ensureFreeSpace(17983696) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:45 INFO MemoryStore: Block broadcast_1028 stored as values in memory (estimated size 17.2 MB, free 3.7 GB)
16/09/07 19:10:46 INFO MemoryStore: ensureFreeSpace(261649) called with curMem=18527520, maxMem=4042889625
16/09/07 19:10:46 INFO MemoryStore: Block broadcast_1028_piece0 stored as bytes in memory (estimated size 255.5 KB, free 3.7 GB)
16/09/07 19:10:46 INFO BlockManagerInfo: Added broadcast_1028_piece0 in memory on localhost:41251 (size: 255.5 KB, free: 3.8 GB)
16/09/07 19:10:46 INFO SparkContext: Created broadcast 1028 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:46 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:46 INFO DAGScheduler: Registering RDD 2083 (repartition at SparkUtils.java:190)
16/09/07 19:10:46 INFO DAGScheduler: Got job 345 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:46 INFO DAGScheduler: Final stage: ResultStage 681(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:46 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 680)
16/09/07 19:10:46 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 680)
16/09/07 19:10:46 INFO DAGScheduler: Submitting ShuffleMapStage 680 (MapPartitionsRDD[2083] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:46 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18789169, maxMem=4042889625
16/09/07 19:10:46 INFO MemoryStore: Block broadcast_1029 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:46 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=18793633, maxMem=4042889625
16/09/07 19:10:46 INFO MemoryStore: Block broadcast_1029_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:46 INFO BlockManagerInfo: Added broadcast_1029_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:46 INFO SparkContext: Created broadcast 1029 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:46 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 680 (MapPartitionsRDD[2083] at repartition at SparkUtils.java:190)
16/09/07 19:10:46 INFO TaskSchedulerImpl: Adding task set 680.0 with 8 tasks
16/09/07 19:10:46 INFO TaskSetManager: Starting task 0.0 in stage 680.0 (TID 5440, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 1.0 in stage 680.0 (TID 5441, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 2.0 in stage 680.0 (TID 5442, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 3.0 in stage 680.0 (TID 5443, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 4.0 in stage 680.0 (TID 5444, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 5.0 in stage 680.0 (TID 5445, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 6.0 in stage 680.0 (TID 5446, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 7.0 in stage 680.0 (TID 5447, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:46 INFO Executor: Running task 1.0 in stage 680.0 (TID 5441)
16/09/07 19:10:46 INFO Executor: Running task 2.0 in stage 680.0 (TID 5442)
16/09/07 19:10:46 INFO Executor: Running task 7.0 in stage 680.0 (TID 5447)
16/09/07 19:10:46 INFO Executor: Running task 3.0 in stage 680.0 (TID 5443)
16/09/07 19:10:46 INFO Executor: Running task 0.0 in stage 680.0 (TID 5440)
16/09/07 19:10:46 INFO Executor: Running task 5.0 in stage 680.0 (TID 5445)
16/09/07 19:10:46 INFO Executor: Running task 4.0 in stage 680.0 (TID 5444)
16/09/07 19:10:46 INFO Executor: Running task 6.0 in stage 680.0 (TID 5446)
16/09/07 19:10:46 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:46 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:46 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:46 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:46 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:46 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:46 INFO Executor: Finished task 1.0 in stage 680.0 (TID 5441). 2260 bytes result sent to driver
16/09/07 19:10:46 INFO Executor: Finished task 7.0 in stage 680.0 (TID 5447). 2260 bytes result sent to driver
16/09/07 19:10:46 INFO Executor: Finished task 5.0 in stage 680.0 (TID 5445). 2260 bytes result sent to driver
16/09/07 19:10:46 INFO TaskSetManager: Finished task 1.0 in stage 680.0 (TID 5441) in 3 ms on localhost (1/8)
16/09/07 19:10:46 INFO TaskSetManager: Finished task 7.0 in stage 680.0 (TID 5447) in 2 ms on localhost (2/8)
16/09/07 19:10:46 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:46 INFO Executor: Finished task 4.0 in stage 680.0 (TID 5444). 2260 bytes result sent to driver
16/09/07 19:10:46 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:46 INFO TaskSetManager: Finished task 5.0 in stage 680.0 (TID 5445) in 3 ms on localhost (3/8)
16/09/07 19:10:46 INFO TaskSetManager: Finished task 4.0 in stage 680.0 (TID 5444) in 3 ms on localhost (4/8)
16/09/07 19:10:46 INFO Executor: Finished task 6.0 in stage 680.0 (TID 5446). 2260 bytes result sent to driver
16/09/07 19:10:46 INFO Executor: Finished task 3.0 in stage 680.0 (TID 5443). 2260 bytes result sent to driver
16/09/07 19:10:46 INFO Executor: Finished task 2.0 in stage 680.0 (TID 5442). 2260 bytes result sent to driver
16/09/07 19:10:46 INFO Executor: Finished task 0.0 in stage 680.0 (TID 5440). 2260 bytes result sent to driver
16/09/07 19:10:46 INFO TaskSetManager: Finished task 6.0 in stage 680.0 (TID 5446) in 4 ms on localhost (5/8)
16/09/07 19:10:46 INFO TaskSetManager: Finished task 3.0 in stage 680.0 (TID 5443) in 4 ms on localhost (6/8)
16/09/07 19:10:46 INFO TaskSetManager: Finished task 2.0 in stage 680.0 (TID 5442) in 5 ms on localhost (7/8)
16/09/07 19:10:46 INFO TaskSetManager: Finished task 0.0 in stage 680.0 (TID 5440) in 5 ms on localhost (8/8)
16/09/07 19:10:46 INFO TaskSchedulerImpl: Removed TaskSet 680.0, whose tasks have all completed, from pool 
16/09/07 19:10:46 INFO DAGScheduler: ShuffleMapStage 680 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:46 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:46 INFO DAGScheduler: running: Set()
16/09/07 19:10:46 INFO DAGScheduler: waiting: Set(ResultStage 681)
16/09/07 19:10:46 INFO DAGScheduler: failed: Set()
16/09/07 19:10:46 INFO DAGScheduler: Missing parents for ResultStage 681: List()
16/09/07 19:10:46 INFO DAGScheduler: Submitting ResultStage 681 (MapPartitionsRDD[2087] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:46 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18796197, maxMem=4042889625
16/09/07 19:10:46 INFO MemoryStore: Block broadcast_1030 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:46 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18801293, maxMem=4042889625
16/09/07 19:10:46 INFO MemoryStore: Block broadcast_1030_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:46 INFO BlockManagerInfo: Added broadcast_1030_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:46 INFO SparkContext: Created broadcast 1030 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:46 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 681 (MapPartitionsRDD[2087] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:46 INFO TaskSchedulerImpl: Adding task set 681.0 with 8 tasks
16/09/07 19:10:46 INFO TaskSetManager: Starting task 0.0 in stage 681.0 (TID 5448, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 1.0 in stage 681.0 (TID 5449, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 2.0 in stage 681.0 (TID 5450, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 3.0 in stage 681.0 (TID 5451, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 4.0 in stage 681.0 (TID 5452, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 5.0 in stage 681.0 (TID 5453, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 6.0 in stage 681.0 (TID 5454, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:46 INFO TaskSetManager: Starting task 7.0 in stage 681.0 (TID 5455, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:46 INFO Executor: Running task 0.0 in stage 681.0 (TID 5448)
16/09/07 19:10:46 INFO Executor: Running task 3.0 in stage 681.0 (TID 5451)
16/09/07 19:10:46 INFO Executor: Running task 1.0 in stage 681.0 (TID 5449)
16/09/07 19:10:46 INFO Executor: Running task 2.0 in stage 681.0 (TID 5450)
16/09/07 19:10:46 INFO Executor: Running task 4.0 in stage 681.0 (TID 5452)
16/09/07 19:10:46 INFO Executor: Running task 6.0 in stage 681.0 (TID 5454)
16/09/07 19:10:46 INFO Executor: Running task 5.0 in stage 681.0 (TID 5453)
16/09/07 19:10:46 INFO Executor: Running task 7.0 in stage 681.0 (TID 5455)
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:46 INFO Executor: Finished task 1.0 in stage 681.0 (TID 5449). 2160 bytes result sent to driver
16/09/07 19:10:46 INFO Executor: Finished task 5.0 in stage 681.0 (TID 5453). 2160 bytes result sent to driver
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:46 INFO TaskSetManager: Finished task 1.0 in stage 681.0 (TID 5449) in 2 ms on localhost (1/8)
16/09/07 19:10:46 INFO TaskSetManager: Finished task 5.0 in stage 681.0 (TID 5453) in 2 ms on localhost (2/8)
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:46 INFO Executor: Finished task 4.0 in stage 681.0 (TID 5452). 2160 bytes result sent to driver
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:46 INFO TaskSetManager: Finished task 4.0 in stage 681.0 (TID 5452) in 2 ms on localhost (3/8)
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:46 INFO Executor: Finished task 2.0 in stage 681.0 (TID 5450). 2160 bytes result sent to driver
16/09/07 19:10:46 INFO TaskSetManager: Finished task 2.0 in stage 681.0 (TID 5450) in 3 ms on localhost (4/8)
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:46 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:46 INFO Executor: Finished task 3.0 in stage 681.0 (TID 5451). 2160 bytes result sent to driver
16/09/07 19:10:46 INFO TaskSetManager: Finished task 3.0 in stage 681.0 (TID 5451) in 3 ms on localhost (5/8)
16/09/07 19:10:46 INFO BlockManagerInfo: Removed broadcast_1029_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:46 INFO Executor: Finished task 7.0 in stage 681.0 (TID 5455). 3485274 bytes result sent to driver
16/09/07 19:10:46 INFO TaskSetManager: Finished task 7.0 in stage 681.0 (TID 5455) in 876 ms on localhost (6/8)
16/09/07 19:10:46 INFO Executor: Finished task 6.0 in stage 681.0 (TID 5454). 3485274 bytes result sent to driver
16/09/07 19:10:46 INFO Executor: Finished task 0.0 in stage 681.0 (TID 5448). 3485274 bytes result sent to driver
16/09/07 19:10:47 INFO TaskSetManager: Finished task 6.0 in stage 681.0 (TID 5454) in 1040 ms on localhost (7/8)
16/09/07 19:10:47 INFO TaskSetManager: Finished task 0.0 in stage 681.0 (TID 5448) in 1040 ms on localhost (8/8)
16/09/07 19:10:47 INFO TaskSchedulerImpl: Removed TaskSet 681.0, whose tasks have all completed, from pool 
16/09/07 19:10:47 INFO DAGScheduler: ResultStage 681 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 1.040 s
16/09/07 19:10:47 INFO DAGScheduler: Job 345 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.049340 s
16/09/07 19:10:47 INFO ParameterAveragingTrainingMaster: Completed training of split 24 of 78
16/09/07 19:10:47 INFO ParameterAveragingTrainingMaster: Starting training of split 25 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:47 INFO MemoryStore: ensureFreeSpace(18313960) called with curMem=18797008, maxMem=4042889625
16/09/07 19:10:47 INFO MemoryStore: Block broadcast_1031 stored as values in memory (estimated size 17.5 MB, free 3.7 GB)
16/09/07 19:10:47 INFO MemoryStore: ensureFreeSpace(225299) called with curMem=37110968, maxMem=4042889625
16/09/07 19:10:47 INFO MemoryStore: Block broadcast_1031_piece0 stored as bytes in memory (estimated size 220.0 KB, free 3.7 GB)
16/09/07 19:10:47 INFO BlockManagerInfo: Added broadcast_1031_piece0 in memory on localhost:41251 (size: 220.0 KB, free: 3.8 GB)
16/09/07 19:10:47 INFO SparkContext: Created broadcast 1031 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:47 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:47 INFO DAGScheduler: Registering RDD 2088 (repartition at SparkUtils.java:190)
16/09/07 19:10:47 INFO DAGScheduler: Got job 346 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:47 INFO DAGScheduler: Final stage: ResultStage 683(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:47 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 682)
16/09/07 19:10:47 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 682)
16/09/07 19:10:47 INFO DAGScheduler: Submitting ShuffleMapStage 682 (MapPartitionsRDD[2088] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:47 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=37336267, maxMem=4042889625
16/09/07 19:10:47 INFO MemoryStore: Block broadcast_1032 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:47 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=37340731, maxMem=4042889625
16/09/07 19:10:47 INFO MemoryStore: Block broadcast_1032_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:47 INFO BlockManagerInfo: Added broadcast_1032_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:47 INFO SparkContext: Created broadcast 1032 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:47 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 682 (MapPartitionsRDD[2088] at repartition at SparkUtils.java:190)
16/09/07 19:10:47 INFO TaskSchedulerImpl: Adding task set 682.0 with 8 tasks
16/09/07 19:10:47 INFO TaskSetManager: Starting task 0.0 in stage 682.0 (TID 5456, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 1.0 in stage 682.0 (TID 5457, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 2.0 in stage 682.0 (TID 5458, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 3.0 in stage 682.0 (TID 5459, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 4.0 in stage 682.0 (TID 5460, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 5.0 in stage 682.0 (TID 5461, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 6.0 in stage 682.0 (TID 5462, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 7.0 in stage 682.0 (TID 5463, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:47 INFO Executor: Running task 0.0 in stage 682.0 (TID 5456)
16/09/07 19:10:47 INFO Executor: Running task 1.0 in stage 682.0 (TID 5457)
16/09/07 19:10:47 INFO Executor: Running task 3.0 in stage 682.0 (TID 5459)
16/09/07 19:10:47 INFO Executor: Running task 2.0 in stage 682.0 (TID 5458)
16/09/07 19:10:47 INFO Executor: Running task 4.0 in stage 682.0 (TID 5460)
16/09/07 19:10:47 INFO Executor: Running task 6.0 in stage 682.0 (TID 5462)
16/09/07 19:10:47 INFO Executor: Running task 5.0 in stage 682.0 (TID 5461)
16/09/07 19:10:47 INFO Executor: Running task 7.0 in stage 682.0 (TID 5463)
16/09/07 19:10:47 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:47 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:47 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:47 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:47 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:47 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:47 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:47 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:47 INFO Executor: Finished task 4.0 in stage 682.0 (TID 5460). 2260 bytes result sent to driver
16/09/07 19:10:47 INFO Executor: Finished task 1.0 in stage 682.0 (TID 5457). 2260 bytes result sent to driver
16/09/07 19:10:47 INFO TaskSetManager: Finished task 4.0 in stage 682.0 (TID 5460) in 2 ms on localhost (1/8)
16/09/07 19:10:47 INFO TaskSetManager: Finished task 1.0 in stage 682.0 (TID 5457) in 2 ms on localhost (2/8)
16/09/07 19:10:47 INFO Executor: Finished task 3.0 in stage 682.0 (TID 5459). 2260 bytes result sent to driver
16/09/07 19:10:47 INFO Executor: Finished task 7.0 in stage 682.0 (TID 5463). 2260 bytes result sent to driver
16/09/07 19:10:47 INFO Executor: Finished task 2.0 in stage 682.0 (TID 5458). 2260 bytes result sent to driver
16/09/07 19:10:47 INFO TaskSetManager: Finished task 3.0 in stage 682.0 (TID 5459) in 3 ms on localhost (3/8)
16/09/07 19:10:47 INFO TaskSetManager: Finished task 7.0 in stage 682.0 (TID 5463) in 3 ms on localhost (4/8)
16/09/07 19:10:47 INFO TaskSetManager: Finished task 2.0 in stage 682.0 (TID 5458) in 3 ms on localhost (5/8)
16/09/07 19:10:47 INFO Executor: Finished task 5.0 in stage 682.0 (TID 5461). 2260 bytes result sent to driver
16/09/07 19:10:47 INFO TaskSetManager: Finished task 5.0 in stage 682.0 (TID 5461) in 3 ms on localhost (6/8)
16/09/07 19:10:47 INFO Executor: Finished task 6.0 in stage 682.0 (TID 5462). 2260 bytes result sent to driver
16/09/07 19:10:47 INFO TaskSetManager: Finished task 6.0 in stage 682.0 (TID 5462) in 3 ms on localhost (7/8)
16/09/07 19:10:47 INFO Executor: Finished task 0.0 in stage 682.0 (TID 5456). 2260 bytes result sent to driver
16/09/07 19:10:47 INFO TaskSetManager: Finished task 0.0 in stage 682.0 (TID 5456) in 4 ms on localhost (8/8)
16/09/07 19:10:47 INFO TaskSchedulerImpl: Removed TaskSet 682.0, whose tasks have all completed, from pool 
16/09/07 19:10:47 INFO DAGScheduler: ShuffleMapStage 682 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:47 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:47 INFO DAGScheduler: running: Set()
16/09/07 19:10:47 INFO DAGScheduler: waiting: Set(ResultStage 683)
16/09/07 19:10:47 INFO DAGScheduler: failed: Set()
16/09/07 19:10:47 INFO DAGScheduler: Missing parents for ResultStage 683: List()
16/09/07 19:10:47 INFO DAGScheduler: Submitting ResultStage 683 (MapPartitionsRDD[2092] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:47 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=37343295, maxMem=4042889625
16/09/07 19:10:47 INFO MemoryStore: Block broadcast_1033 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:47 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=37348391, maxMem=4042889625
16/09/07 19:10:47 INFO MemoryStore: Block broadcast_1033_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:47 INFO BlockManagerInfo: Added broadcast_1033_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:47 INFO SparkContext: Created broadcast 1033 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:47 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 683 (MapPartitionsRDD[2092] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:47 INFO TaskSchedulerImpl: Adding task set 683.0 with 8 tasks
16/09/07 19:10:47 INFO TaskSetManager: Starting task 0.0 in stage 683.0 (TID 5464, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 1.0 in stage 683.0 (TID 5465, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 2.0 in stage 683.0 (TID 5466, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 3.0 in stage 683.0 (TID 5467, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 4.0 in stage 683.0 (TID 5468, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 5.0 in stage 683.0 (TID 5469, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 6.0 in stage 683.0 (TID 5470, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:47 INFO TaskSetManager: Starting task 7.0 in stage 683.0 (TID 5471, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:47 INFO Executor: Running task 0.0 in stage 683.0 (TID 5464)
16/09/07 19:10:47 INFO Executor: Running task 3.0 in stage 683.0 (TID 5467)
16/09/07 19:10:47 INFO Executor: Running task 1.0 in stage 683.0 (TID 5465)
16/09/07 19:10:47 INFO Executor: Running task 6.0 in stage 683.0 (TID 5470)
16/09/07 19:10:47 INFO Executor: Running task 2.0 in stage 683.0 (TID 5466)
16/09/07 19:10:47 INFO BlockManagerInfo: Removed broadcast_1032_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:47 INFO Executor: Running task 4.0 in stage 683.0 (TID 5468)
16/09/07 19:10:47 INFO Executor: Running task 5.0 in stage 683.0 (TID 5469)
16/09/07 19:10:47 INFO Executor: Running task 7.0 in stage 683.0 (TID 5471)
16/09/07 19:10:47 INFO BlockManagerInfo: Removed broadcast_1030_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:47 INFO ContextCleaner: Cleaned accumulator 682
16/09/07 19:10:47 INFO ContextCleaner: Cleaned accumulator 681
16/09/07 19:10:47 INFO BlockManagerInfo: Removed broadcast_1028_piece0 on localhost:41251 in memory (size: 255.5 KB, free: 3.8 GB)
16/09/07 19:10:47 INFO ContextCleaner: Cleaned shuffle 335
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:47 INFO Executor: Finished task 3.0 in stage 683.0 (TID 5467). 2160 bytes result sent to driver
16/09/07 19:10:47 INFO TaskSetManager: Finished task 3.0 in stage 683.0 (TID 5467) in 84 ms on localhost (1/8)
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:47 INFO Executor: Finished task 2.0 in stage 683.0 (TID 5466). 2160 bytes result sent to driver
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:47 INFO TaskSetManager: Finished task 2.0 in stage 683.0 (TID 5466) in 84 ms on localhost (2/8)
16/09/07 19:10:47 INFO Executor: Finished task 5.0 in stage 683.0 (TID 5469). 2160 bytes result sent to driver
16/09/07 19:10:47 INFO TaskSetManager: Finished task 5.0 in stage 683.0 (TID 5469) in 85 ms on localhost (3/8)
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:47 INFO Executor: Finished task 4.0 in stage 683.0 (TID 5468). 2160 bytes result sent to driver
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:47 INFO Executor: Finished task 1.0 in stage 683.0 (TID 5465). 2160 bytes result sent to driver
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:47 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:47 INFO TaskSetManager: Finished task 4.0 in stage 683.0 (TID 5468) in 86 ms on localhost (4/8)
16/09/07 19:10:47 INFO TaskSetManager: Finished task 1.0 in stage 683.0 (TID 5465) in 86 ms on localhost (5/8)
16/09/07 19:10:48 INFO Executor: Finished task 0.0 in stage 683.0 (TID 5464). 3485274 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 0.0 in stage 683.0 (TID 5464) in 886 ms on localhost (6/8)
16/09/07 19:10:48 INFO Executor: Finished task 7.0 in stage 683.0 (TID 5471). 3485274 bytes result sent to driver
16/09/07 19:10:48 INFO Executor: Finished task 6.0 in stage 683.0 (TID 5470). 3485274 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 7.0 in stage 683.0 (TID 5471) in 971 ms on localhost (7/8)
16/09/07 19:10:48 INFO TaskSetManager: Finished task 6.0 in stage 683.0 (TID 5470) in 1052 ms on localhost (8/8)
16/09/07 19:10:48 INFO TaskSchedulerImpl: Removed TaskSet 683.0, whose tasks have all completed, from pool 
16/09/07 19:10:48 INFO DAGScheduler: ResultStage 683 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 1.053 s
16/09/07 19:10:48 INFO DAGScheduler: Job 346 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.060560 s
16/09/07 19:10:48 INFO ParameterAveragingTrainingMaster: Completed training of split 25 of 78
16/09/07 19:10:48 INFO ParameterAveragingTrainingMaster: Starting training of split 26 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:48 INFO MemoryStore: ensureFreeSpace(18322648) called with curMem=19090922, maxMem=4042889625
16/09/07 19:10:48 INFO MemoryStore: Block broadcast_1034 stored as values in memory (estimated size 17.5 MB, free 3.7 GB)
16/09/07 19:10:48 INFO MemoryStore: ensureFreeSpace(195483) called with curMem=37413570, maxMem=4042889625
16/09/07 19:10:48 INFO MemoryStore: Block broadcast_1034_piece0 stored as bytes in memory (estimated size 190.9 KB, free 3.7 GB)
16/09/07 19:10:48 INFO BlockManagerInfo: Added broadcast_1034_piece0 in memory on localhost:41251 (size: 190.9 KB, free: 3.8 GB)
16/09/07 19:10:48 INFO SparkContext: Created broadcast 1034 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:48 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:48 INFO DAGScheduler: Registering RDD 2093 (repartition at SparkUtils.java:190)
16/09/07 19:10:48 INFO DAGScheduler: Got job 347 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:48 INFO DAGScheduler: Final stage: ResultStage 685(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:48 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 684)
16/09/07 19:10:48 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 684)
16/09/07 19:10:48 INFO DAGScheduler: Submitting ShuffleMapStage 684 (MapPartitionsRDD[2093] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:48 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=37609053, maxMem=4042889625
16/09/07 19:10:48 INFO MemoryStore: Block broadcast_1035 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:48 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=37613517, maxMem=4042889625
16/09/07 19:10:48 INFO MemoryStore: Block broadcast_1035_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:48 INFO BlockManagerInfo: Added broadcast_1035_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:48 INFO SparkContext: Created broadcast 1035 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:48 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 684 (MapPartitionsRDD[2093] at repartition at SparkUtils.java:190)
16/09/07 19:10:48 INFO TaskSchedulerImpl: Adding task set 684.0 with 8 tasks
16/09/07 19:10:48 INFO TaskSetManager: Starting task 0.0 in stage 684.0 (TID 5472, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 1.0 in stage 684.0 (TID 5473, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 2.0 in stage 684.0 (TID 5474, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 3.0 in stage 684.0 (TID 5475, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 4.0 in stage 684.0 (TID 5476, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 5.0 in stage 684.0 (TID 5477, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 6.0 in stage 684.0 (TID 5478, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 7.0 in stage 684.0 (TID 5479, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:48 INFO Executor: Running task 1.0 in stage 684.0 (TID 5473)
16/09/07 19:10:48 INFO Executor: Running task 0.0 in stage 684.0 (TID 5472)
16/09/07 19:10:48 INFO Executor: Running task 6.0 in stage 684.0 (TID 5478)
16/09/07 19:10:48 INFO Executor: Running task 4.0 in stage 684.0 (TID 5476)
16/09/07 19:10:48 INFO Executor: Running task 5.0 in stage 684.0 (TID 5477)
16/09/07 19:10:48 INFO Executor: Running task 2.0 in stage 684.0 (TID 5474)
16/09/07 19:10:48 INFO Executor: Running task 7.0 in stage 684.0 (TID 5479)
16/09/07 19:10:48 INFO Executor: Running task 3.0 in stage 684.0 (TID 5475)
16/09/07 19:10:48 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:48 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:48 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:48 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:48 INFO Executor: Finished task 1.0 in stage 684.0 (TID 5473). 2260 bytes result sent to driver
16/09/07 19:10:48 INFO Executor: Finished task 4.0 in stage 684.0 (TID 5476). 2260 bytes result sent to driver
16/09/07 19:10:48 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:48 INFO TaskSetManager: Finished task 1.0 in stage 684.0 (TID 5473) in 2 ms on localhost (1/8)
16/09/07 19:10:48 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:48 INFO Executor: Finished task 2.0 in stage 684.0 (TID 5474). 2260 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 4.0 in stage 684.0 (TID 5476) in 2 ms on localhost (2/8)
16/09/07 19:10:48 INFO TaskSetManager: Finished task 2.0 in stage 684.0 (TID 5474) in 2 ms on localhost (3/8)
16/09/07 19:10:48 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:48 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:48 INFO Executor: Finished task 0.0 in stage 684.0 (TID 5472). 2260 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 0.0 in stage 684.0 (TID 5472) in 3 ms on localhost (4/8)
16/09/07 19:10:48 INFO Executor: Finished task 6.0 in stage 684.0 (TID 5478). 2260 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 6.0 in stage 684.0 (TID 5478) in 4 ms on localhost (5/8)
16/09/07 19:10:48 INFO Executor: Finished task 3.0 in stage 684.0 (TID 5475). 2260 bytes result sent to driver
16/09/07 19:10:48 INFO Executor: Finished task 5.0 in stage 684.0 (TID 5477). 2260 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 3.0 in stage 684.0 (TID 5475) in 4 ms on localhost (6/8)
16/09/07 19:10:48 INFO Executor: Finished task 7.0 in stage 684.0 (TID 5479). 2260 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 5.0 in stage 684.0 (TID 5477) in 4 ms on localhost (7/8)
16/09/07 19:10:48 INFO TaskSetManager: Finished task 7.0 in stage 684.0 (TID 5479) in 4 ms on localhost (8/8)
16/09/07 19:10:48 INFO TaskSchedulerImpl: Removed TaskSet 684.0, whose tasks have all completed, from pool 
16/09/07 19:10:48 INFO DAGScheduler: ShuffleMapStage 684 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:48 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:48 INFO DAGScheduler: running: Set()
16/09/07 19:10:48 INFO DAGScheduler: waiting: Set(ResultStage 685)
16/09/07 19:10:48 INFO DAGScheduler: failed: Set()
16/09/07 19:10:48 INFO DAGScheduler: Missing parents for ResultStage 685: List()
16/09/07 19:10:48 INFO DAGScheduler: Submitting ResultStage 685 (MapPartitionsRDD[2097] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:48 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=37616081, maxMem=4042889625
16/09/07 19:10:48 INFO MemoryStore: Block broadcast_1036 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:48 INFO MemoryStore: ensureFreeSpace(2740) called with curMem=37621177, maxMem=4042889625
16/09/07 19:10:48 INFO MemoryStore: Block broadcast_1036_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:48 INFO BlockManagerInfo: Added broadcast_1036_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:48 INFO SparkContext: Created broadcast 1036 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:48 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 685 (MapPartitionsRDD[2097] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:48 INFO TaskSchedulerImpl: Adding task set 685.0 with 8 tasks
16/09/07 19:10:48 INFO TaskSetManager: Starting task 0.0 in stage 685.0 (TID 5480, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 1.0 in stage 685.0 (TID 5481, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 2.0 in stage 685.0 (TID 5482, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 3.0 in stage 685.0 (TID 5483, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 4.0 in stage 685.0 (TID 5484, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 5.0 in stage 685.0 (TID 5485, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 6.0 in stage 685.0 (TID 5486, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:48 INFO TaskSetManager: Starting task 7.0 in stage 685.0 (TID 5487, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:48 INFO Executor: Running task 1.0 in stage 685.0 (TID 5481)
16/09/07 19:10:48 INFO Executor: Running task 2.0 in stage 685.0 (TID 5482)
16/09/07 19:10:48 INFO Executor: Running task 6.0 in stage 685.0 (TID 5486)
16/09/07 19:10:48 INFO Executor: Running task 0.0 in stage 685.0 (TID 5480)
16/09/07 19:10:48 INFO Executor: Running task 3.0 in stage 685.0 (TID 5483)
16/09/07 19:10:48 INFO Executor: Running task 7.0 in stage 685.0 (TID 5487)
16/09/07 19:10:48 INFO Executor: Running task 5.0 in stage 685.0 (TID 5485)
16/09/07 19:10:48 INFO Executor: Running task 4.0 in stage 685.0 (TID 5484)
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:48 INFO Executor: Finished task 3.0 in stage 685.0 (TID 5483). 2160 bytes result sent to driver
16/09/07 19:10:48 INFO Executor: Finished task 1.0 in stage 685.0 (TID 5481). 2160 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 3.0 in stage 685.0 (TID 5483) in 2 ms on localhost (1/8)
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:48 INFO TaskSetManager: Finished task 1.0 in stage 685.0 (TID 5481) in 2 ms on localhost (2/8)
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:48 INFO Executor: Finished task 2.0 in stage 685.0 (TID 5482). 2160 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 2.0 in stage 685.0 (TID 5482) in 3 ms on localhost (3/8)
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:48 INFO Executor: Finished task 4.0 in stage 685.0 (TID 5484). 2160 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 4.0 in stage 685.0 (TID 5484) in 4 ms on localhost (4/8)
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:48 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:48 INFO Executor: Finished task 5.0 in stage 685.0 (TID 5485). 2160 bytes result sent to driver
16/09/07 19:10:48 INFO TaskSetManager: Finished task 5.0 in stage 685.0 (TID 5485) in 5 ms on localhost (5/8)
16/09/07 19:10:48 INFO BlockManagerInfo: Removed broadcast_1035_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:48 INFO BlockManagerInfo: Removed broadcast_1033_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:48 INFO ContextCleaner: Cleaned accumulator 684
16/09/07 19:10:48 INFO ContextCleaner: Cleaned accumulator 683
16/09/07 19:10:48 INFO BlockManagerInfo: Removed broadcast_1031_piece0 on localhost:41251 in memory (size: 220.0 KB, free: 3.8 GB)
16/09/07 19:10:48 INFO ContextCleaner: Cleaned shuffle 336
16/09/07 19:10:49 INFO Executor: Finished task 0.0 in stage 685.0 (TID 5480). 3485274 bytes result sent to driver
16/09/07 19:10:49 INFO Executor: Finished task 6.0 in stage 685.0 (TID 5486). 3485274 bytes result sent to driver
16/09/07 19:10:49 INFO Executor: Finished task 7.0 in stage 685.0 (TID 5487). 3485274 bytes result sent to driver
16/09/07 19:10:49 INFO TaskSetManager: Finished task 0.0 in stage 685.0 (TID 5480) in 849 ms on localhost (6/8)
16/09/07 19:10:49 INFO TaskSetManager: Finished task 6.0 in stage 685.0 (TID 5486) in 861 ms on localhost (7/8)
16/09/07 19:10:49 INFO TaskSetManager: Finished task 7.0 in stage 685.0 (TID 5487) in 863 ms on localhost (8/8)
16/09/07 19:10:49 INFO TaskSchedulerImpl: Removed TaskSet 685.0, whose tasks have all completed, from pool 
16/09/07 19:10:49 INFO DAGScheduler: ResultStage 685 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.863 s
16/09/07 19:10:49 INFO DAGScheduler: Job 347 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.872039 s
16/09/07 19:10:49 INFO ParameterAveragingTrainingMaster: Completed training of split 26 of 78
16/09/07 19:10:49 INFO ParameterAveragingTrainingMaster: Starting training of split 27 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:49 INFO BlockManagerInfo: Removed broadcast_1036_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:49 INFO ContextCleaner: Cleaned accumulator 686
16/09/07 19:10:49 INFO ContextCleaner: Cleaned accumulator 685
16/09/07 19:10:49 INFO BlockManagerInfo: Removed broadcast_1034_piece0 on localhost:41251 in memory (size: 190.9 KB, free: 3.8 GB)
16/09/07 19:10:49 INFO ContextCleaner: Cleaned shuffle 337
16/09/07 19:10:49 INFO MemoryStore: ensureFreeSpace(18319312) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:49 INFO MemoryStore: Block broadcast_1037 stored as values in memory (estimated size 17.5 MB, free 3.7 GB)
16/09/07 19:10:49 INFO MemoryStore: ensureFreeSpace(177108) called with curMem=18863136, maxMem=4042889625
16/09/07 19:10:49 INFO MemoryStore: Block broadcast_1037_piece0 stored as bytes in memory (estimated size 173.0 KB, free 3.7 GB)
16/09/07 19:10:49 INFO BlockManagerInfo: Added broadcast_1037_piece0 in memory on localhost:41251 (size: 173.0 KB, free: 3.8 GB)
16/09/07 19:10:49 INFO SparkContext: Created broadcast 1037 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:49 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:49 INFO DAGScheduler: Registering RDD 2098 (repartition at SparkUtils.java:190)
16/09/07 19:10:49 INFO DAGScheduler: Got job 348 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:49 INFO DAGScheduler: Final stage: ResultStage 687(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:49 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 686)
16/09/07 19:10:49 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 686)
16/09/07 19:10:49 INFO DAGScheduler: Submitting ShuffleMapStage 686 (MapPartitionsRDD[2098] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:49 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19040244, maxMem=4042889625
16/09/07 19:10:49 INFO MemoryStore: Block broadcast_1038 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:49 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19044708, maxMem=4042889625
16/09/07 19:10:49 INFO MemoryStore: Block broadcast_1038_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:49 INFO BlockManagerInfo: Added broadcast_1038_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:49 INFO SparkContext: Created broadcast 1038 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:49 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 686 (MapPartitionsRDD[2098] at repartition at SparkUtils.java:190)
16/09/07 19:10:49 INFO TaskSchedulerImpl: Adding task set 686.0 with 8 tasks
16/09/07 19:10:49 INFO TaskSetManager: Starting task 0.0 in stage 686.0 (TID 5488, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 1.0 in stage 686.0 (TID 5489, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 2.0 in stage 686.0 (TID 5490, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 3.0 in stage 686.0 (TID 5491, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 4.0 in stage 686.0 (TID 5492, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 5.0 in stage 686.0 (TID 5493, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 6.0 in stage 686.0 (TID 5494, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 7.0 in stage 686.0 (TID 5495, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:49 INFO Executor: Running task 2.0 in stage 686.0 (TID 5490)
16/09/07 19:10:49 INFO Executor: Running task 0.0 in stage 686.0 (TID 5488)
16/09/07 19:10:49 INFO Executor: Running task 7.0 in stage 686.0 (TID 5495)
16/09/07 19:10:49 INFO Executor: Running task 3.0 in stage 686.0 (TID 5491)
16/09/07 19:10:49 INFO Executor: Running task 1.0 in stage 686.0 (TID 5489)
16/09/07 19:10:49 INFO Executor: Running task 6.0 in stage 686.0 (TID 5494)
16/09/07 19:10:49 INFO Executor: Running task 5.0 in stage 686.0 (TID 5493)
16/09/07 19:10:49 INFO Executor: Running task 4.0 in stage 686.0 (TID 5492)
16/09/07 19:10:49 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:49 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:49 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:49 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:49 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:49 INFO Executor: Finished task 0.0 in stage 686.0 (TID 5488). 2260 bytes result sent to driver
16/09/07 19:10:49 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:49 INFO Executor: Finished task 6.0 in stage 686.0 (TID 5494). 2260 bytes result sent to driver
16/09/07 19:10:49 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:49 INFO Executor: Finished task 2.0 in stage 686.0 (TID 5490). 2260 bytes result sent to driver
16/09/07 19:10:49 INFO TaskSetManager: Finished task 0.0 in stage 686.0 (TID 5488) in 2 ms on localhost (1/8)
16/09/07 19:10:49 INFO Executor: Finished task 7.0 in stage 686.0 (TID 5495). 2260 bytes result sent to driver
16/09/07 19:10:49 INFO TaskSetManager: Finished task 6.0 in stage 686.0 (TID 5494) in 2 ms on localhost (2/8)
16/09/07 19:10:49 INFO TaskSetManager: Finished task 2.0 in stage 686.0 (TID 5490) in 2 ms on localhost (3/8)
16/09/07 19:10:49 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:49 INFO TaskSetManager: Finished task 7.0 in stage 686.0 (TID 5495) in 2 ms on localhost (4/8)
16/09/07 19:10:49 INFO Executor: Finished task 3.0 in stage 686.0 (TID 5491). 2260 bytes result sent to driver
16/09/07 19:10:49 INFO TaskSetManager: Finished task 3.0 in stage 686.0 (TID 5491) in 3 ms on localhost (5/8)
16/09/07 19:10:49 INFO Executor: Finished task 4.0 in stage 686.0 (TID 5492). 2260 bytes result sent to driver
16/09/07 19:10:49 INFO Executor: Finished task 1.0 in stage 686.0 (TID 5489). 2260 bytes result sent to driver
16/09/07 19:10:49 INFO Executor: Finished task 5.0 in stage 686.0 (TID 5493). 2260 bytes result sent to driver
16/09/07 19:10:49 INFO TaskSetManager: Finished task 4.0 in stage 686.0 (TID 5492) in 3 ms on localhost (6/8)
16/09/07 19:10:49 INFO TaskSetManager: Finished task 1.0 in stage 686.0 (TID 5489) in 4 ms on localhost (7/8)
16/09/07 19:10:49 INFO TaskSetManager: Finished task 5.0 in stage 686.0 (TID 5493) in 4 ms on localhost (8/8)
16/09/07 19:10:49 INFO TaskSchedulerImpl: Removed TaskSet 686.0, whose tasks have all completed, from pool 
16/09/07 19:10:49 INFO DAGScheduler: ShuffleMapStage 686 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:49 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:49 INFO DAGScheduler: running: Set()
16/09/07 19:10:49 INFO DAGScheduler: waiting: Set(ResultStage 687)
16/09/07 19:10:49 INFO DAGScheduler: failed: Set()
16/09/07 19:10:49 INFO DAGScheduler: Missing parents for ResultStage 687: List()
16/09/07 19:10:49 INFO DAGScheduler: Submitting ResultStage 687 (MapPartitionsRDD[2102] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:49 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19047272, maxMem=4042889625
16/09/07 19:10:49 INFO MemoryStore: Block broadcast_1039 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:49 INFO MemoryStore: ensureFreeSpace(2741) called with curMem=19052368, maxMem=4042889625
16/09/07 19:10:49 INFO MemoryStore: Block broadcast_1039_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:49 INFO BlockManagerInfo: Added broadcast_1039_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:49 INFO SparkContext: Created broadcast 1039 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:49 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 687 (MapPartitionsRDD[2102] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:49 INFO TaskSchedulerImpl: Adding task set 687.0 with 8 tasks
16/09/07 19:10:49 INFO TaskSetManager: Starting task 0.0 in stage 687.0 (TID 5496, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 1.0 in stage 687.0 (TID 5497, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 2.0 in stage 687.0 (TID 5498, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 3.0 in stage 687.0 (TID 5499, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 4.0 in stage 687.0 (TID 5500, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 5.0 in stage 687.0 (TID 5501, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 6.0 in stage 687.0 (TID 5502, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:49 INFO TaskSetManager: Starting task 7.0 in stage 687.0 (TID 5503, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:49 INFO Executor: Running task 2.0 in stage 687.0 (TID 5498)
16/09/07 19:10:49 INFO Executor: Running task 4.0 in stage 687.0 (TID 5500)
16/09/07 19:10:49 INFO Executor: Running task 5.0 in stage 687.0 (TID 5501)
16/09/07 19:10:49 INFO Executor: Running task 0.0 in stage 687.0 (TID 5496)
16/09/07 19:10:49 INFO Executor: Running task 3.0 in stage 687.0 (TID 5499)
16/09/07 19:10:49 INFO Executor: Running task 6.0 in stage 687.0 (TID 5502)
16/09/07 19:10:49 INFO Executor: Running task 1.0 in stage 687.0 (TID 5497)
16/09/07 19:10:49 INFO Executor: Running task 7.0 in stage 687.0 (TID 5503)
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:49 INFO Executor: Finished task 4.0 in stage 687.0 (TID 5500). 2160 bytes result sent to driver
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:49 INFO TaskSetManager: Finished task 4.0 in stage 687.0 (TID 5500) in 1 ms on localhost (1/8)
16/09/07 19:10:49 INFO Executor: Finished task 1.0 in stage 687.0 (TID 5497). 2160 bytes result sent to driver
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:49 INFO Executor: Finished task 3.0 in stage 687.0 (TID 5499). 2160 bytes result sent to driver
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:49 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:49 INFO Executor: Finished task 5.0 in stage 687.0 (TID 5501). 2160 bytes result sent to driver
16/09/07 19:10:49 INFO Executor: Finished task 2.0 in stage 687.0 (TID 5498). 2160 bytes result sent to driver
16/09/07 19:10:49 INFO TaskSetManager: Finished task 1.0 in stage 687.0 (TID 5497) in 4 ms on localhost (2/8)
16/09/07 19:10:49 INFO TaskSetManager: Finished task 3.0 in stage 687.0 (TID 5499) in 3 ms on localhost (3/8)
16/09/07 19:10:49 INFO TaskSetManager: Finished task 5.0 in stage 687.0 (TID 5501) in 3 ms on localhost (4/8)
16/09/07 19:10:49 INFO TaskSetManager: Finished task 2.0 in stage 687.0 (TID 5498) in 5 ms on localhost (5/8)
16/09/07 19:10:50 INFO BlockManagerInfo: Removed broadcast_1038_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:50 INFO Executor: Finished task 7.0 in stage 687.0 (TID 5503). 3485274 bytes result sent to driver
16/09/07 19:10:50 INFO Executor: Finished task 0.0 in stage 687.0 (TID 5496). 3485274 bytes result sent to driver
16/09/07 19:10:50 INFO Executor: Finished task 6.0 in stage 687.0 (TID 5502). 3485274 bytes result sent to driver
16/09/07 19:10:50 INFO TaskSetManager: Finished task 7.0 in stage 687.0 (TID 5503) in 975 ms on localhost (6/8)
16/09/07 19:10:50 INFO TaskSetManager: Finished task 0.0 in stage 687.0 (TID 5496) in 976 ms on localhost (7/8)
16/09/07 19:10:50 INFO TaskSetManager: Finished task 6.0 in stage 687.0 (TID 5502) in 995 ms on localhost (8/8)
16/09/07 19:10:50 INFO DAGScheduler: ResultStage 687 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.996 s
16/09/07 19:10:50 INFO TaskSchedulerImpl: Removed TaskSet 687.0, whose tasks have all completed, from pool 
16/09/07 19:10:50 INFO DAGScheduler: Job 348 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.003733 s
16/09/07 19:10:50 INFO ParameterAveragingTrainingMaster: Completed training of split 27 of 78
16/09/07 19:10:50 INFO ParameterAveragingTrainingMaster: Starting training of split 28 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:50 INFO BlockManagerInfo: Removed broadcast_1039_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:50 INFO ContextCleaner: Cleaned accumulator 688
16/09/07 19:10:50 INFO ContextCleaner: Cleaned accumulator 687
16/09/07 19:10:50 INFO BlockManagerInfo: Removed broadcast_1037_piece0 on localhost:41251 in memory (size: 173.0 KB, free: 3.8 GB)
16/09/07 19:10:50 INFO ContextCleaner: Cleaned shuffle 338
16/09/07 19:10:51 INFO MemoryStore: ensureFreeSpace(18115936) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:51 INFO MemoryStore: Block broadcast_1040 stored as values in memory (estimated size 17.3 MB, free 3.7 GB)
16/09/07 19:10:51 INFO MemoryStore: ensureFreeSpace(170869) called with curMem=18659760, maxMem=4042889625
16/09/07 19:10:51 INFO MemoryStore: Block broadcast_1040_piece0 stored as bytes in memory (estimated size 166.9 KB, free 3.7 GB)
16/09/07 19:10:51 INFO BlockManagerInfo: Added broadcast_1040_piece0 in memory on localhost:41251 (size: 166.9 KB, free: 3.8 GB)
16/09/07 19:10:51 INFO SparkContext: Created broadcast 1040 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:51 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:51 INFO DAGScheduler: Registering RDD 2103 (repartition at SparkUtils.java:190)
16/09/07 19:10:51 INFO DAGScheduler: Got job 349 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:51 INFO DAGScheduler: Final stage: ResultStage 689(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:51 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 688)
16/09/07 19:10:51 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 688)
16/09/07 19:10:51 INFO DAGScheduler: Submitting ShuffleMapStage 688 (MapPartitionsRDD[2103] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:51 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18830629, maxMem=4042889625
16/09/07 19:10:51 INFO MemoryStore: Block broadcast_1041 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:51 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=18835093, maxMem=4042889625
16/09/07 19:10:51 INFO MemoryStore: Block broadcast_1041_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:51 INFO BlockManagerInfo: Added broadcast_1041_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:51 INFO SparkContext: Created broadcast 1041 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:51 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 688 (MapPartitionsRDD[2103] at repartition at SparkUtils.java:190)
16/09/07 19:10:51 INFO TaskSchedulerImpl: Adding task set 688.0 with 8 tasks
16/09/07 19:10:51 INFO TaskSetManager: Starting task 0.0 in stage 688.0 (TID 5504, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 1.0 in stage 688.0 (TID 5505, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 2.0 in stage 688.0 (TID 5506, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 3.0 in stage 688.0 (TID 5507, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 4.0 in stage 688.0 (TID 5508, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 5.0 in stage 688.0 (TID 5509, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 6.0 in stage 688.0 (TID 5510, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 7.0 in stage 688.0 (TID 5511, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:51 INFO Executor: Running task 2.0 in stage 688.0 (TID 5506)
16/09/07 19:10:51 INFO Executor: Running task 5.0 in stage 688.0 (TID 5509)
16/09/07 19:10:51 INFO Executor: Running task 3.0 in stage 688.0 (TID 5507)
16/09/07 19:10:51 INFO Executor: Running task 4.0 in stage 688.0 (TID 5508)
16/09/07 19:10:51 INFO Executor: Running task 7.0 in stage 688.0 (TID 5511)
16/09/07 19:10:51 INFO Executor: Running task 0.0 in stage 688.0 (TID 5504)
16/09/07 19:10:51 INFO Executor: Running task 1.0 in stage 688.0 (TID 5505)
16/09/07 19:10:51 INFO Executor: Running task 6.0 in stage 688.0 (TID 5510)
16/09/07 19:10:51 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:51 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:51 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:51 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:51 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:51 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:51 INFO Executor: Finished task 4.0 in stage 688.0 (TID 5508). 2260 bytes result sent to driver
16/09/07 19:10:51 INFO Executor: Finished task 2.0 in stage 688.0 (TID 5506). 2260 bytes result sent to driver
16/09/07 19:10:51 INFO Executor: Finished task 1.0 in stage 688.0 (TID 5505). 2260 bytes result sent to driver
16/09/07 19:10:51 INFO Executor: Finished task 3.0 in stage 688.0 (TID 5507). 2260 bytes result sent to driver
16/09/07 19:10:51 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:51 INFO TaskSetManager: Finished task 2.0 in stage 688.0 (TID 5506) in 3 ms on localhost (1/8)
16/09/07 19:10:51 INFO TaskSetManager: Finished task 4.0 in stage 688.0 (TID 5508) in 3 ms on localhost (2/8)
16/09/07 19:10:51 INFO TaskSetManager: Finished task 1.0 in stage 688.0 (TID 5505) in 3 ms on localhost (3/8)
16/09/07 19:10:51 INFO TaskSetManager: Finished task 3.0 in stage 688.0 (TID 5507) in 3 ms on localhost (4/8)
16/09/07 19:10:51 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:51 INFO Executor: Finished task 0.0 in stage 688.0 (TID 5504). 2260 bytes result sent to driver
16/09/07 19:10:51 INFO TaskSetManager: Finished task 0.0 in stage 688.0 (TID 5504) in 4 ms on localhost (5/8)
16/09/07 19:10:51 INFO Executor: Finished task 5.0 in stage 688.0 (TID 5509). 2260 bytes result sent to driver
16/09/07 19:10:51 INFO Executor: Finished task 6.0 in stage 688.0 (TID 5510). 2260 bytes result sent to driver
16/09/07 19:10:51 INFO Executor: Finished task 7.0 in stage 688.0 (TID 5511). 2260 bytes result sent to driver
16/09/07 19:10:51 INFO TaskSetManager: Finished task 6.0 in stage 688.0 (TID 5510) in 5 ms on localhost (6/8)
16/09/07 19:10:51 INFO TaskSetManager: Finished task 5.0 in stage 688.0 (TID 5509) in 5 ms on localhost (7/8)
16/09/07 19:10:51 INFO TaskSetManager: Finished task 7.0 in stage 688.0 (TID 5511) in 5 ms on localhost (8/8)
16/09/07 19:10:51 INFO TaskSchedulerImpl: Removed TaskSet 688.0, whose tasks have all completed, from pool 
16/09/07 19:10:51 INFO DAGScheduler: ShuffleMapStage 688 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:51 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:51 INFO DAGScheduler: running: Set()
16/09/07 19:10:51 INFO DAGScheduler: waiting: Set(ResultStage 689)
16/09/07 19:10:51 INFO DAGScheduler: failed: Set()
16/09/07 19:10:51 INFO DAGScheduler: Missing parents for ResultStage 689: List()
16/09/07 19:10:51 INFO DAGScheduler: Submitting ResultStage 689 (MapPartitionsRDD[2107] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:51 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18837656, maxMem=4042889625
16/09/07 19:10:51 INFO MemoryStore: Block broadcast_1042 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:51 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18842752, maxMem=4042889625
16/09/07 19:10:51 INFO MemoryStore: Block broadcast_1042_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:51 INFO BlockManagerInfo: Added broadcast_1042_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:51 INFO SparkContext: Created broadcast 1042 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:51 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 689 (MapPartitionsRDD[2107] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:51 INFO TaskSchedulerImpl: Adding task set 689.0 with 8 tasks
16/09/07 19:10:51 INFO TaskSetManager: Starting task 0.0 in stage 689.0 (TID 5512, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 1.0 in stage 689.0 (TID 5513, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 2.0 in stage 689.0 (TID 5514, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 3.0 in stage 689.0 (TID 5515, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 4.0 in stage 689.0 (TID 5516, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 5.0 in stage 689.0 (TID 5517, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 6.0 in stage 689.0 (TID 5518, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:51 INFO TaskSetManager: Starting task 7.0 in stage 689.0 (TID 5519, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:51 INFO Executor: Running task 0.0 in stage 689.0 (TID 5512)
16/09/07 19:10:51 INFO Executor: Running task 1.0 in stage 689.0 (TID 5513)
16/09/07 19:10:51 INFO Executor: Running task 7.0 in stage 689.0 (TID 5519)
16/09/07 19:10:51 INFO Executor: Running task 2.0 in stage 689.0 (TID 5514)
16/09/07 19:10:51 INFO Executor: Running task 5.0 in stage 689.0 (TID 5517)
16/09/07 19:10:51 INFO Executor: Running task 3.0 in stage 689.0 (TID 5515)
16/09/07 19:10:51 INFO Executor: Running task 4.0 in stage 689.0 (TID 5516)
16/09/07 19:10:51 INFO Executor: Running task 6.0 in stage 689.0 (TID 5518)
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:51 INFO Executor: Finished task 2.0 in stage 689.0 (TID 5514). 2160 bytes result sent to driver
16/09/07 19:10:51 INFO Executor: Finished task 3.0 in stage 689.0 (TID 5515). 2160 bytes result sent to driver
16/09/07 19:10:51 INFO TaskSetManager: Finished task 2.0 in stage 689.0 (TID 5514) in 2 ms on localhost (1/8)
16/09/07 19:10:51 INFO TaskSetManager: Finished task 3.0 in stage 689.0 (TID 5515) in 1 ms on localhost (2/8)
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:51 INFO Executor: Finished task 4.0 in stage 689.0 (TID 5516). 2160 bytes result sent to driver
16/09/07 19:10:51 INFO TaskSetManager: Finished task 4.0 in stage 689.0 (TID 5516) in 3 ms on localhost (3/8)
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:51 INFO Executor: Finished task 1.0 in stage 689.0 (TID 5513). 2160 bytes result sent to driver
16/09/07 19:10:51 INFO Executor: Finished task 5.0 in stage 689.0 (TID 5517). 2160 bytes result sent to driver
16/09/07 19:10:51 INFO TaskSetManager: Finished task 1.0 in stage 689.0 (TID 5513) in 5 ms on localhost (4/8)
16/09/07 19:10:51 INFO TaskSetManager: Finished task 5.0 in stage 689.0 (TID 5517) in 4 ms on localhost (5/8)
16/09/07 19:10:51 INFO BlockManagerInfo: Removed broadcast_1041_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:52 INFO Executor: Finished task 7.0 in stage 689.0 (TID 5519). 3485274 bytes result sent to driver
16/09/07 19:10:52 INFO Executor: Finished task 6.0 in stage 689.0 (TID 5518). 3485274 bytes result sent to driver
16/09/07 19:10:52 INFO Executor: Finished task 0.0 in stage 689.0 (TID 5512). 3485274 bytes result sent to driver
16/09/07 19:10:52 INFO TaskSetManager: Finished task 6.0 in stage 689.0 (TID 5518) in 904 ms on localhost (6/8)
16/09/07 19:10:52 INFO TaskSetManager: Finished task 7.0 in stage 689.0 (TID 5519) in 915 ms on localhost (7/8)
16/09/07 19:10:52 INFO TaskSetManager: Finished task 0.0 in stage 689.0 (TID 5512) in 917 ms on localhost (8/8)
16/09/07 19:10:52 INFO TaskSchedulerImpl: Removed TaskSet 689.0, whose tasks have all completed, from pool 
16/09/07 19:10:52 INFO DAGScheduler: ResultStage 689 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.917 s
16/09/07 19:10:52 INFO DAGScheduler: Job 349 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.925777 s
16/09/07 19:10:52 INFO ParameterAveragingTrainingMaster: Completed training of split 28 of 78
16/09/07 19:10:52 INFO ParameterAveragingTrainingMaster: Starting training of split 29 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:52 INFO MemoryStore: ensureFreeSpace(18357192) called with curMem=18838468, maxMem=4042889625
16/09/07 19:10:52 INFO MemoryStore: Block broadcast_1043 stored as values in memory (estimated size 17.5 MB, free 3.7 GB)
16/09/07 19:10:52 INFO BlockManagerInfo: Removed broadcast_1042_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:52 INFO ContextCleaner: Cleaned accumulator 690
16/09/07 19:10:52 INFO ContextCleaner: Cleaned accumulator 689
16/09/07 19:10:52 INFO BlockManagerInfo: Removed broadcast_1040_piece0 on localhost:41251 in memory (size: 166.9 KB, free: 3.8 GB)
16/09/07 19:10:52 INFO ContextCleaner: Cleaned shuffle 339
16/09/07 19:10:52 INFO MemoryStore: ensureFreeSpace(169384) called with curMem=18901016, maxMem=4042889625
16/09/07 19:10:52 INFO MemoryStore: Block broadcast_1043_piece0 stored as bytes in memory (estimated size 165.4 KB, free 3.7 GB)
16/09/07 19:10:52 INFO BlockManagerInfo: Added broadcast_1043_piece0 in memory on localhost:41251 (size: 165.4 KB, free: 3.8 GB)
16/09/07 19:10:52 INFO SparkContext: Created broadcast 1043 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:52 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:52 INFO DAGScheduler: Registering RDD 2108 (repartition at SparkUtils.java:190)
16/09/07 19:10:52 INFO DAGScheduler: Got job 350 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:52 INFO DAGScheduler: Final stage: ResultStage 691(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:52 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 690)
16/09/07 19:10:52 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 690)
16/09/07 19:10:52 INFO DAGScheduler: Submitting ShuffleMapStage 690 (MapPartitionsRDD[2108] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:52 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19070400, maxMem=4042889625
16/09/07 19:10:52 INFO MemoryStore: Block broadcast_1044 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:52 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19074864, maxMem=4042889625
16/09/07 19:10:52 INFO MemoryStore: Block broadcast_1044_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:52 INFO BlockManagerInfo: Added broadcast_1044_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:52 INFO SparkContext: Created broadcast 1044 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:52 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 690 (MapPartitionsRDD[2108] at repartition at SparkUtils.java:190)
16/09/07 19:10:52 INFO TaskSchedulerImpl: Adding task set 690.0 with 8 tasks
16/09/07 19:10:52 INFO TaskSetManager: Starting task 0.0 in stage 690.0 (TID 5520, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 1.0 in stage 690.0 (TID 5521, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 2.0 in stage 690.0 (TID 5522, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 3.0 in stage 690.0 (TID 5523, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 4.0 in stage 690.0 (TID 5524, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 5.0 in stage 690.0 (TID 5525, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 6.0 in stage 690.0 (TID 5526, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 7.0 in stage 690.0 (TID 5527, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:52 INFO Executor: Running task 1.0 in stage 690.0 (TID 5521)
16/09/07 19:10:52 INFO Executor: Running task 3.0 in stage 690.0 (TID 5523)
16/09/07 19:10:52 INFO Executor: Running task 5.0 in stage 690.0 (TID 5525)
16/09/07 19:10:52 INFO Executor: Running task 2.0 in stage 690.0 (TID 5522)
16/09/07 19:10:52 INFO Executor: Running task 0.0 in stage 690.0 (TID 5520)
16/09/07 19:10:52 INFO Executor: Running task 4.0 in stage 690.0 (TID 5524)
16/09/07 19:10:52 INFO Executor: Running task 7.0 in stage 690.0 (TID 5527)
16/09/07 19:10:52 INFO Executor: Running task 6.0 in stage 690.0 (TID 5526)
16/09/07 19:10:52 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:52 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:52 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:52 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:52 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:52 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:52 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:52 INFO Executor: Finished task 2.0 in stage 690.0 (TID 5522). 2260 bytes result sent to driver
16/09/07 19:10:52 INFO Executor: Finished task 0.0 in stage 690.0 (TID 5520). 2260 bytes result sent to driver
16/09/07 19:10:52 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:52 INFO Executor: Finished task 7.0 in stage 690.0 (TID 5527). 2260 bytes result sent to driver
16/09/07 19:10:52 INFO TaskSetManager: Finished task 0.0 in stage 690.0 (TID 5520) in 2 ms on localhost (1/8)
16/09/07 19:10:52 INFO TaskSetManager: Finished task 2.0 in stage 690.0 (TID 5522) in 2 ms on localhost (2/8)
16/09/07 19:10:52 INFO Executor: Finished task 6.0 in stage 690.0 (TID 5526). 2260 bytes result sent to driver
16/09/07 19:10:52 INFO Executor: Finished task 4.0 in stage 690.0 (TID 5524). 2260 bytes result sent to driver
16/09/07 19:10:52 INFO TaskSetManager: Finished task 7.0 in stage 690.0 (TID 5527) in 3 ms on localhost (3/8)
16/09/07 19:10:52 INFO TaskSetManager: Finished task 4.0 in stage 690.0 (TID 5524) in 3 ms on localhost (4/8)
16/09/07 19:10:52 INFO TaskSetManager: Finished task 6.0 in stage 690.0 (TID 5526) in 3 ms on localhost (5/8)
16/09/07 19:10:52 INFO Executor: Finished task 3.0 in stage 690.0 (TID 5523). 2260 bytes result sent to driver
16/09/07 19:10:52 INFO Executor: Finished task 1.0 in stage 690.0 (TID 5521). 2260 bytes result sent to driver
16/09/07 19:10:52 INFO TaskSetManager: Finished task 3.0 in stage 690.0 (TID 5523) in 3 ms on localhost (6/8)
16/09/07 19:10:52 INFO Executor: Finished task 5.0 in stage 690.0 (TID 5525). 2260 bytes result sent to driver
16/09/07 19:10:52 INFO TaskSetManager: Finished task 1.0 in stage 690.0 (TID 5521) in 3 ms on localhost (7/8)
16/09/07 19:10:52 INFO TaskSetManager: Finished task 5.0 in stage 690.0 (TID 5525) in 3 ms on localhost (8/8)
16/09/07 19:10:52 INFO TaskSchedulerImpl: Removed TaskSet 690.0, whose tasks have all completed, from pool 
16/09/07 19:10:52 INFO DAGScheduler: ShuffleMapStage 690 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:52 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:52 INFO DAGScheduler: running: Set()
16/09/07 19:10:52 INFO DAGScheduler: waiting: Set(ResultStage 691)
16/09/07 19:10:52 INFO DAGScheduler: failed: Set()
16/09/07 19:10:52 INFO DAGScheduler: Missing parents for ResultStage 691: List()
16/09/07 19:10:52 INFO DAGScheduler: Submitting ResultStage 691 (MapPartitionsRDD[2112] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:52 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19077428, maxMem=4042889625
16/09/07 19:10:52 INFO MemoryStore: Block broadcast_1045 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:52 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19082524, maxMem=4042889625
16/09/07 19:10:52 INFO MemoryStore: Block broadcast_1045_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:52 INFO BlockManagerInfo: Added broadcast_1045_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:52 INFO SparkContext: Created broadcast 1045 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:52 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 691 (MapPartitionsRDD[2112] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:52 INFO TaskSchedulerImpl: Adding task set 691.0 with 8 tasks
16/09/07 19:10:52 INFO TaskSetManager: Starting task 0.0 in stage 691.0 (TID 5528, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 1.0 in stage 691.0 (TID 5529, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 2.0 in stage 691.0 (TID 5530, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 3.0 in stage 691.0 (TID 5531, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 4.0 in stage 691.0 (TID 5532, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 5.0 in stage 691.0 (TID 5533, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 6.0 in stage 691.0 (TID 5534, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:52 INFO TaskSetManager: Starting task 7.0 in stage 691.0 (TID 5535, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:52 INFO Executor: Running task 2.0 in stage 691.0 (TID 5530)
16/09/07 19:10:52 INFO Executor: Running task 4.0 in stage 691.0 (TID 5532)
16/09/07 19:10:52 INFO Executor: Running task 6.0 in stage 691.0 (TID 5534)
16/09/07 19:10:52 INFO Executor: Running task 1.0 in stage 691.0 (TID 5529)
16/09/07 19:10:52 INFO Executor: Running task 0.0 in stage 691.0 (TID 5528)
16/09/07 19:10:52 INFO Executor: Running task 3.0 in stage 691.0 (TID 5531)
16/09/07 19:10:52 INFO Executor: Running task 7.0 in stage 691.0 (TID 5535)
16/09/07 19:10:52 INFO Executor: Running task 5.0 in stage 691.0 (TID 5533)
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:52 INFO Executor: Finished task 2.0 in stage 691.0 (TID 5530). 2160 bytes result sent to driver
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:52 INFO Executor: Finished task 5.0 in stage 691.0 (TID 5533). 2160 bytes result sent to driver
16/09/07 19:10:52 INFO TaskSetManager: Finished task 5.0 in stage 691.0 (TID 5533) in 1 ms on localhost (1/8)
16/09/07 19:10:52 INFO TaskSetManager: Finished task 2.0 in stage 691.0 (TID 5530) in 1 ms on localhost (2/8)
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:52 INFO Executor: Finished task 1.0 in stage 691.0 (TID 5529). 2160 bytes result sent to driver
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:52 INFO Executor: Finished task 3.0 in stage 691.0 (TID 5531). 2160 bytes result sent to driver
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:52 INFO TaskSetManager: Finished task 1.0 in stage 691.0 (TID 5529) in 2 ms on localhost (3/8)
16/09/07 19:10:52 INFO TaskSetManager: Finished task 3.0 in stage 691.0 (TID 5531) in 2 ms on localhost (4/8)
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:52 INFO Executor: Finished task 4.0 in stage 691.0 (TID 5532). 2160 bytes result sent to driver
16/09/07 19:10:52 INFO TaskSetManager: Finished task 4.0 in stage 691.0 (TID 5532) in 3 ms on localhost (5/8)
16/09/07 19:10:52 INFO BlockManagerInfo: Removed broadcast_1044_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:53 INFO Executor: Finished task 0.0 in stage 691.0 (TID 5528). 3485274 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 0.0 in stage 691.0 (TID 5528) in 926 ms on localhost (6/8)
16/09/07 19:10:53 INFO Executor: Finished task 6.0 in stage 691.0 (TID 5534). 3485274 bytes result sent to driver
16/09/07 19:10:53 INFO Executor: Finished task 7.0 in stage 691.0 (TID 5535). 3485274 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 6.0 in stage 691.0 (TID 5534) in 969 ms on localhost (7/8)
16/09/07 19:10:53 INFO TaskSetManager: Finished task 7.0 in stage 691.0 (TID 5535) in 980 ms on localhost (8/8)
16/09/07 19:10:53 INFO TaskSchedulerImpl: Removed TaskSet 691.0, whose tasks have all completed, from pool 
16/09/07 19:10:53 INFO DAGScheduler: ResultStage 691 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.980 s
16/09/07 19:10:53 INFO DAGScheduler: Job 350 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.988050 s
16/09/07 19:10:53 INFO ParameterAveragingTrainingMaster: Completed training of split 29 of 78
16/09/07 19:10:53 INFO ParameterAveragingTrainingMaster: Starting training of split 30 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:53 INFO BlockManagerInfo: Removed broadcast_1045_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:53 INFO ContextCleaner: Cleaned accumulator 692
16/09/07 19:10:53 INFO ContextCleaner: Cleaned accumulator 691
16/09/07 19:10:53 INFO BlockManagerInfo: Removed broadcast_1043_piece0 on localhost:41251 in memory (size: 165.4 KB, free: 3.8 GB)
16/09/07 19:10:53 INFO ContextCleaner: Cleaned shuffle 340
16/09/07 19:10:53 INFO MemoryStore: ensureFreeSpace(18182168) called with curMem=543824, maxMem=4042889625
16/09/07 19:10:53 INFO MemoryStore: Block broadcast_1046 stored as values in memory (estimated size 17.3 MB, free 3.7 GB)
16/09/07 19:10:53 INFO MemoryStore: ensureFreeSpace(168691) called with curMem=18725992, maxMem=4042889625
16/09/07 19:10:53 INFO MemoryStore: Block broadcast_1046_piece0 stored as bytes in memory (estimated size 164.7 KB, free 3.7 GB)
16/09/07 19:10:53 INFO BlockManagerInfo: Added broadcast_1046_piece0 in memory on localhost:41251 (size: 164.7 KB, free: 3.8 GB)
16/09/07 19:10:53 INFO SparkContext: Created broadcast 1046 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:53 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:53 INFO DAGScheduler: Registering RDD 2113 (repartition at SparkUtils.java:190)
16/09/07 19:10:53 INFO DAGScheduler: Got job 351 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:53 INFO DAGScheduler: Final stage: ResultStage 693(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:53 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 692)
16/09/07 19:10:53 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 692)
16/09/07 19:10:53 INFO DAGScheduler: Submitting ShuffleMapStage 692 (MapPartitionsRDD[2113] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:53 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=18894683, maxMem=4042889625
16/09/07 19:10:53 INFO MemoryStore: Block broadcast_1047 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:53 INFO MemoryStore: ensureFreeSpace(2566) called with curMem=18899147, maxMem=4042889625
16/09/07 19:10:53 INFO MemoryStore: Block broadcast_1047_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:53 INFO BlockManagerInfo: Added broadcast_1047_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:53 INFO SparkContext: Created broadcast 1047 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:53 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 692 (MapPartitionsRDD[2113] at repartition at SparkUtils.java:190)
16/09/07 19:10:53 INFO TaskSchedulerImpl: Adding task set 692.0 with 8 tasks
16/09/07 19:10:53 INFO TaskSetManager: Starting task 0.0 in stage 692.0 (TID 5536, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 1.0 in stage 692.0 (TID 5537, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 2.0 in stage 692.0 (TID 5538, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 3.0 in stage 692.0 (TID 5539, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 4.0 in stage 692.0 (TID 5540, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 5.0 in stage 692.0 (TID 5541, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 6.0 in stage 692.0 (TID 5542, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 7.0 in stage 692.0 (TID 5543, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:53 INFO Executor: Running task 1.0 in stage 692.0 (TID 5537)
16/09/07 19:10:53 INFO Executor: Running task 2.0 in stage 692.0 (TID 5538)
16/09/07 19:10:53 INFO Executor: Running task 3.0 in stage 692.0 (TID 5539)
16/09/07 19:10:53 INFO Executor: Running task 4.0 in stage 692.0 (TID 5540)
16/09/07 19:10:53 INFO Executor: Running task 0.0 in stage 692.0 (TID 5536)
16/09/07 19:10:53 INFO Executor: Running task 5.0 in stage 692.0 (TID 5541)
16/09/07 19:10:53 INFO Executor: Running task 6.0 in stage 692.0 (TID 5542)
16/09/07 19:10:53 INFO Executor: Running task 7.0 in stage 692.0 (TID 5543)
16/09/07 19:10:53 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:53 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:53 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:53 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:53 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:53 INFO Executor: Finished task 1.0 in stage 692.0 (TID 5537). 2260 bytes result sent to driver
16/09/07 19:10:53 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:53 INFO TaskSetManager: Finished task 1.0 in stage 692.0 (TID 5537) in 3 ms on localhost (1/8)
16/09/07 19:10:53 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:53 INFO Executor: Finished task 4.0 in stage 692.0 (TID 5540). 2260 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 4.0 in stage 692.0 (TID 5540) in 2 ms on localhost (2/8)
16/09/07 19:10:53 INFO Executor: Finished task 7.0 in stage 692.0 (TID 5543). 2260 bytes result sent to driver
16/09/07 19:10:53 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:53 INFO TaskSetManager: Finished task 7.0 in stage 692.0 (TID 5543) in 3 ms on localhost (3/8)
16/09/07 19:10:53 INFO Executor: Finished task 6.0 in stage 692.0 (TID 5542). 2260 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 6.0 in stage 692.0 (TID 5542) in 3 ms on localhost (4/8)
16/09/07 19:10:53 INFO Executor: Finished task 0.0 in stage 692.0 (TID 5536). 2260 bytes result sent to driver
16/09/07 19:10:53 INFO Executor: Finished task 5.0 in stage 692.0 (TID 5541). 2260 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 0.0 in stage 692.0 (TID 5536) in 4 ms on localhost (5/8)
16/09/07 19:10:53 INFO TaskSetManager: Finished task 5.0 in stage 692.0 (TID 5541) in 3 ms on localhost (6/8)
16/09/07 19:10:53 INFO Executor: Finished task 2.0 in stage 692.0 (TID 5538). 2260 bytes result sent to driver
16/09/07 19:10:53 INFO Executor: Finished task 3.0 in stage 692.0 (TID 5539). 2260 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 2.0 in stage 692.0 (TID 5538) in 5 ms on localhost (7/8)
16/09/07 19:10:53 INFO TaskSetManager: Finished task 3.0 in stage 692.0 (TID 5539) in 4 ms on localhost (8/8)
16/09/07 19:10:53 INFO TaskSchedulerImpl: Removed TaskSet 692.0, whose tasks have all completed, from pool 
16/09/07 19:10:53 INFO DAGScheduler: ShuffleMapStage 692 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:10:53 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:53 INFO DAGScheduler: running: Set()
16/09/07 19:10:53 INFO DAGScheduler: waiting: Set(ResultStage 693)
16/09/07 19:10:53 INFO DAGScheduler: failed: Set()
16/09/07 19:10:53 INFO DAGScheduler: Missing parents for ResultStage 693: List()
16/09/07 19:10:53 INFO DAGScheduler: Submitting ResultStage 693 (MapPartitionsRDD[2117] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:53 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=18901713, maxMem=4042889625
16/09/07 19:10:53 INFO MemoryStore: Block broadcast_1048 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:53 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=18906809, maxMem=4042889625
16/09/07 19:10:53 INFO MemoryStore: Block broadcast_1048_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:53 INFO BlockManagerInfo: Added broadcast_1048_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:53 INFO SparkContext: Created broadcast 1048 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:53 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 693 (MapPartitionsRDD[2117] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:53 INFO TaskSchedulerImpl: Adding task set 693.0 with 8 tasks
16/09/07 19:10:53 INFO TaskSetManager: Starting task 0.0 in stage 693.0 (TID 5544, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 1.0 in stage 693.0 (TID 5545, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 2.0 in stage 693.0 (TID 5546, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 3.0 in stage 693.0 (TID 5547, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 4.0 in stage 693.0 (TID 5548, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 5.0 in stage 693.0 (TID 5549, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 6.0 in stage 693.0 (TID 5550, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:53 INFO TaskSetManager: Starting task 7.0 in stage 693.0 (TID 5551, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:53 INFO Executor: Running task 1.0 in stage 693.0 (TID 5545)
16/09/07 19:10:53 INFO Executor: Running task 7.0 in stage 693.0 (TID 5551)
16/09/07 19:10:53 INFO Executor: Running task 4.0 in stage 693.0 (TID 5548)
16/09/07 19:10:53 INFO Executor: Running task 3.0 in stage 693.0 (TID 5547)
16/09/07 19:10:53 INFO Executor: Running task 0.0 in stage 693.0 (TID 5544)
16/09/07 19:10:53 INFO Executor: Running task 5.0 in stage 693.0 (TID 5549)
16/09/07 19:10:53 INFO Executor: Running task 2.0 in stage 693.0 (TID 5546)
16/09/07 19:10:53 INFO Executor: Running task 6.0 in stage 693.0 (TID 5550)
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:53 INFO Executor: Finished task 2.0 in stage 693.0 (TID 5546). 2160 bytes result sent to driver
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:53 INFO TaskSetManager: Finished task 2.0 in stage 693.0 (TID 5546) in 1 ms on localhost (1/8)
16/09/07 19:10:53 INFO Executor: Finished task 5.0 in stage 693.0 (TID 5549). 2160 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 5.0 in stage 693.0 (TID 5549) in 2 ms on localhost (2/8)
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:53 INFO Executor: Finished task 3.0 in stage 693.0 (TID 5547). 2160 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 3.0 in stage 693.0 (TID 5547) in 3 ms on localhost (3/8)
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:53 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:53 INFO Executor: Finished task 1.0 in stage 693.0 (TID 5545). 2160 bytes result sent to driver
16/09/07 19:10:53 INFO Executor: Finished task 4.0 in stage 693.0 (TID 5548). 2160 bytes result sent to driver
16/09/07 19:10:53 INFO TaskSetManager: Finished task 1.0 in stage 693.0 (TID 5545) in 4 ms on localhost (4/8)
16/09/07 19:10:53 INFO TaskSetManager: Finished task 4.0 in stage 693.0 (TID 5548) in 4 ms on localhost (5/8)
16/09/07 19:10:53 INFO BlockManagerInfo: Removed broadcast_1047_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:54 INFO Executor: Finished task 7.0 in stage 693.0 (TID 5551). 3485274 bytes result sent to driver
16/09/07 19:10:54 INFO Executor: Finished task 0.0 in stage 693.0 (TID 5544). 3485274 bytes result sent to driver
16/09/07 19:10:54 INFO TaskSetManager: Finished task 7.0 in stage 693.0 (TID 5551) in 956 ms on localhost (6/8)
16/09/07 19:10:54 INFO Executor: Finished task 6.0 in stage 693.0 (TID 5550). 3485274 bytes result sent to driver
16/09/07 19:10:54 INFO TaskSetManager: Finished task 0.0 in stage 693.0 (TID 5544) in 973 ms on localhost (7/8)
16/09/07 19:10:54 INFO TaskSetManager: Finished task 6.0 in stage 693.0 (TID 5550) in 984 ms on localhost (8/8)
16/09/07 19:10:54 INFO TaskSchedulerImpl: Removed TaskSet 693.0, whose tasks have all completed, from pool 
16/09/07 19:10:54 INFO DAGScheduler: ResultStage 693 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.986 s
16/09/07 19:10:54 INFO DAGScheduler: Job 351 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.993769 s
16/09/07 19:10:54 INFO ParameterAveragingTrainingMaster: Completed training of split 30 of 78
16/09/07 19:10:54 INFO ParameterAveragingTrainingMaster: Starting training of split 31 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:54 INFO MemoryStore: ensureFreeSpace(18378392) called with curMem=18902522, maxMem=4042889625
16/09/07 19:10:54 INFO MemoryStore: Block broadcast_1049 stored as values in memory (estimated size 17.5 MB, free 3.7 GB)
16/09/07 19:10:55 INFO BlockManagerInfo: Removed broadcast_1048_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:55 INFO ContextCleaner: Cleaned accumulator 694
16/09/07 19:10:55 INFO ContextCleaner: Cleaned accumulator 693
16/09/07 19:10:55 INFO BlockManagerInfo: Removed broadcast_1046_piece0 on localhost:41251 in memory (size: 164.7 KB, free: 3.8 GB)
16/09/07 19:10:55 INFO ContextCleaner: Cleaned shuffle 341
16/09/07 19:10:55 INFO MemoryStore: ensureFreeSpace(168593) called with curMem=18922216, maxMem=4042889625
16/09/07 19:10:55 INFO MemoryStore: Block broadcast_1049_piece0 stored as bytes in memory (estimated size 164.6 KB, free 3.7 GB)
16/09/07 19:10:55 INFO BlockManagerInfo: Added broadcast_1049_piece0 in memory on localhost:41251 (size: 164.6 KB, free: 3.8 GB)
16/09/07 19:10:55 INFO SparkContext: Created broadcast 1049 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:55 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:55 INFO DAGScheduler: Registering RDD 2118 (repartition at SparkUtils.java:190)
16/09/07 19:10:55 INFO DAGScheduler: Got job 352 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:55 INFO DAGScheduler: Final stage: ResultStage 695(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:55 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 694)
16/09/07 19:10:55 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 694)
16/09/07 19:10:55 INFO DAGScheduler: Submitting ShuffleMapStage 694 (MapPartitionsRDD[2118] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:55 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19090809, maxMem=4042889625
16/09/07 19:10:55 INFO MemoryStore: Block broadcast_1050 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:55 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19095273, maxMem=4042889625
16/09/07 19:10:55 INFO MemoryStore: Block broadcast_1050_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:55 INFO BlockManagerInfo: Added broadcast_1050_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:55 INFO SparkContext: Created broadcast 1050 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:55 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 694 (MapPartitionsRDD[2118] at repartition at SparkUtils.java:190)
16/09/07 19:10:55 INFO TaskSchedulerImpl: Adding task set 694.0 with 8 tasks
16/09/07 19:10:55 INFO TaskSetManager: Starting task 0.0 in stage 694.0 (TID 5552, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 1.0 in stage 694.0 (TID 5553, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 2.0 in stage 694.0 (TID 5554, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 3.0 in stage 694.0 (TID 5555, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 4.0 in stage 694.0 (TID 5556, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 5.0 in stage 694.0 (TID 5557, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 6.0 in stage 694.0 (TID 5558, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 7.0 in stage 694.0 (TID 5559, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:55 INFO Executor: Running task 3.0 in stage 694.0 (TID 5555)
16/09/07 19:10:55 INFO Executor: Running task 0.0 in stage 694.0 (TID 5552)
16/09/07 19:10:55 INFO Executor: Running task 6.0 in stage 694.0 (TID 5558)
16/09/07 19:10:55 INFO Executor: Running task 5.0 in stage 694.0 (TID 5557)
16/09/07 19:10:55 INFO Executor: Running task 2.0 in stage 694.0 (TID 5554)
16/09/07 19:10:55 INFO Executor: Running task 7.0 in stage 694.0 (TID 5559)
16/09/07 19:10:55 INFO Executor: Running task 1.0 in stage 694.0 (TID 5553)
16/09/07 19:10:55 INFO Executor: Running task 4.0 in stage 694.0 (TID 5556)
16/09/07 19:10:55 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:55 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:55 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:55 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:55 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:55 INFO Executor: Finished task 3.0 in stage 694.0 (TID 5555). 2260 bytes result sent to driver
16/09/07 19:10:55 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:55 INFO TaskSetManager: Finished task 3.0 in stage 694.0 (TID 5555) in 2 ms on localhost (1/8)
16/09/07 19:10:55 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:55 INFO Executor: Finished task 4.0 in stage 694.0 (TID 5556). 2260 bytes result sent to driver
16/09/07 19:10:55 INFO Executor: Finished task 5.0 in stage 694.0 (TID 5557). 2260 bytes result sent to driver
16/09/07 19:10:55 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:55 INFO Executor: Finished task 6.0 in stage 694.0 (TID 5558). 2260 bytes result sent to driver
16/09/07 19:10:55 INFO TaskSetManager: Finished task 4.0 in stage 694.0 (TID 5556) in 3 ms on localhost (2/8)
16/09/07 19:10:55 INFO TaskSetManager: Finished task 5.0 in stage 694.0 (TID 5557) in 3 ms on localhost (3/8)
16/09/07 19:10:55 INFO TaskSetManager: Finished task 6.0 in stage 694.0 (TID 5558) in 3 ms on localhost (4/8)
16/09/07 19:10:55 INFO Executor: Finished task 2.0 in stage 694.0 (TID 5554). 2260 bytes result sent to driver
16/09/07 19:10:55 INFO TaskSetManager: Finished task 2.0 in stage 694.0 (TID 5554) in 3 ms on localhost (5/8)
16/09/07 19:10:55 INFO Executor: Finished task 0.0 in stage 694.0 (TID 5552). 2260 bytes result sent to driver
16/09/07 19:10:55 INFO Executor: Finished task 7.0 in stage 694.0 (TID 5559). 2260 bytes result sent to driver
16/09/07 19:10:55 INFO TaskSetManager: Finished task 0.0 in stage 694.0 (TID 5552) in 3 ms on localhost (6/8)
16/09/07 19:10:55 INFO Executor: Finished task 1.0 in stage 694.0 (TID 5553). 2260 bytes result sent to driver
16/09/07 19:10:55 INFO TaskSetManager: Finished task 7.0 in stage 694.0 (TID 5559) in 4 ms on localhost (7/8)
16/09/07 19:10:55 INFO TaskSetManager: Finished task 1.0 in stage 694.0 (TID 5553) in 4 ms on localhost (8/8)
16/09/07 19:10:55 INFO TaskSchedulerImpl: Removed TaskSet 694.0, whose tasks have all completed, from pool 
16/09/07 19:10:55 INFO DAGScheduler: ShuffleMapStage 694 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:55 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:55 INFO DAGScheduler: running: Set()
16/09/07 19:10:55 INFO DAGScheduler: waiting: Set(ResultStage 695)
16/09/07 19:10:55 INFO DAGScheduler: failed: Set()
16/09/07 19:10:55 INFO DAGScheduler: Missing parents for ResultStage 695: List()
16/09/07 19:10:55 INFO DAGScheduler: Submitting ResultStage 695 (MapPartitionsRDD[2122] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:55 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19097837, maxMem=4042889625
16/09/07 19:10:55 INFO MemoryStore: Block broadcast_1051 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:55 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19102933, maxMem=4042889625
16/09/07 19:10:55 INFO MemoryStore: Block broadcast_1051_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:55 INFO BlockManagerInfo: Added broadcast_1051_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:55 INFO SparkContext: Created broadcast 1051 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:55 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 695 (MapPartitionsRDD[2122] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:55 INFO TaskSchedulerImpl: Adding task set 695.0 with 8 tasks
16/09/07 19:10:55 INFO TaskSetManager: Starting task 0.0 in stage 695.0 (TID 5560, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 1.0 in stage 695.0 (TID 5561, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 2.0 in stage 695.0 (TID 5562, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 3.0 in stage 695.0 (TID 5563, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 4.0 in stage 695.0 (TID 5564, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 5.0 in stage 695.0 (TID 5565, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 6.0 in stage 695.0 (TID 5566, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:55 INFO TaskSetManager: Starting task 7.0 in stage 695.0 (TID 5567, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:55 INFO Executor: Running task 0.0 in stage 695.0 (TID 5560)
16/09/07 19:10:55 INFO Executor: Running task 2.0 in stage 695.0 (TID 5562)
16/09/07 19:10:55 INFO Executor: Running task 4.0 in stage 695.0 (TID 5564)
16/09/07 19:10:55 INFO Executor: Running task 7.0 in stage 695.0 (TID 5567)
16/09/07 19:10:55 INFO Executor: Running task 3.0 in stage 695.0 (TID 5563)
16/09/07 19:10:55 INFO Executor: Running task 1.0 in stage 695.0 (TID 5561)
16/09/07 19:10:55 INFO Executor: Running task 6.0 in stage 695.0 (TID 5566)
16/09/07 19:10:55 INFO Executor: Running task 5.0 in stage 695.0 (TID 5565)
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:55 INFO Executor: Finished task 5.0 in stage 695.0 (TID 5565). 2160 bytes result sent to driver
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:55 INFO TaskSetManager: Finished task 5.0 in stage 695.0 (TID 5565) in 1 ms on localhost (1/8)
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:55 INFO Executor: Finished task 3.0 in stage 695.0 (TID 5563). 2160 bytes result sent to driver
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:55 INFO Executor: Finished task 1.0 in stage 695.0 (TID 5561). 2160 bytes result sent to driver
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:55 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:55 INFO TaskSetManager: Finished task 3.0 in stage 695.0 (TID 5563) in 3 ms on localhost (2/8)
16/09/07 19:10:55 INFO TaskSetManager: Finished task 1.0 in stage 695.0 (TID 5561) in 3 ms on localhost (3/8)
16/09/07 19:10:55 INFO Executor: Finished task 4.0 in stage 695.0 (TID 5564). 2160 bytes result sent to driver
16/09/07 19:10:55 INFO Executor: Finished task 2.0 in stage 695.0 (TID 5562). 2160 bytes result sent to driver
16/09/07 19:10:55 INFO TaskSetManager: Finished task 4.0 in stage 695.0 (TID 5564) in 3 ms on localhost (4/8)
16/09/07 19:10:55 INFO TaskSetManager: Finished task 2.0 in stage 695.0 (TID 5562) in 3 ms on localhost (5/8)
16/09/07 19:10:55 INFO BlockManagerInfo: Removed broadcast_1050_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:55 INFO Executor: Finished task 0.0 in stage 695.0 (TID 5560). 3485274 bytes result sent to driver
16/09/07 19:10:55 INFO TaskSetManager: Finished task 0.0 in stage 695.0 (TID 5560) in 849 ms on localhost (6/8)
16/09/07 19:10:55 INFO Executor: Finished task 7.0 in stage 695.0 (TID 5567). 3485274 bytes result sent to driver
16/09/07 19:10:55 INFO Executor: Finished task 6.0 in stage 695.0 (TID 5566). 3485274 bytes result sent to driver
16/09/07 19:10:55 INFO TaskSetManager: Finished task 7.0 in stage 695.0 (TID 5567) in 897 ms on localhost (7/8)
16/09/07 19:10:55 INFO TaskSetManager: Finished task 6.0 in stage 695.0 (TID 5566) in 899 ms on localhost (8/8)
16/09/07 19:10:55 INFO DAGScheduler: ResultStage 695 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.900 s
16/09/07 19:10:55 INFO TaskSchedulerImpl: Removed TaskSet 695.0, whose tasks have all completed, from pool 
16/09/07 19:10:55 INFO DAGScheduler: Job 352 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.907593 s
16/09/07 19:10:55 INFO ParameterAveragingTrainingMaster: Completed training of split 31 of 78
16/09/07 19:10:55 INFO ParameterAveragingTrainingMaster: Starting training of split 32 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:56 INFO MemoryStore: ensureFreeSpace(18407504) called with curMem=19098648, maxMem=4042889625
16/09/07 19:10:56 INFO MemoryStore: Block broadcast_1052 stored as values in memory (estimated size 17.6 MB, free 3.7 GB)
16/09/07 19:10:56 INFO BlockManagerInfo: Removed broadcast_1051_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:56 INFO ContextCleaner: Cleaned accumulator 696
16/09/07 19:10:56 INFO ContextCleaner: Cleaned accumulator 695
16/09/07 19:10:56 INFO BlockManagerInfo: Removed broadcast_1049_piece0 on localhost:41251 in memory (size: 164.6 KB, free: 3.8 GB)
16/09/07 19:10:56 INFO ContextCleaner: Cleaned shuffle 342
16/09/07 19:10:56 INFO MemoryStore: ensureFreeSpace(168625) called with curMem=18951328, maxMem=4042889625
16/09/07 19:10:56 INFO MemoryStore: Block broadcast_1052_piece0 stored as bytes in memory (estimated size 164.7 KB, free 3.7 GB)
16/09/07 19:10:56 INFO BlockManagerInfo: Added broadcast_1052_piece0 in memory on localhost:41251 (size: 164.7 KB, free: 3.8 GB)
16/09/07 19:10:56 INFO SparkContext: Created broadcast 1052 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:56 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:56 INFO DAGScheduler: Registering RDD 2123 (repartition at SparkUtils.java:190)
16/09/07 19:10:56 INFO DAGScheduler: Got job 353 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:56 INFO DAGScheduler: Final stage: ResultStage 697(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:56 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 696)
16/09/07 19:10:56 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 696)
16/09/07 19:10:56 INFO DAGScheduler: Submitting ShuffleMapStage 696 (MapPartitionsRDD[2123] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:56 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19119953, maxMem=4042889625
16/09/07 19:10:56 INFO MemoryStore: Block broadcast_1053 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:56 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=19124417, maxMem=4042889625
16/09/07 19:10:56 INFO MemoryStore: Block broadcast_1053_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:56 INFO BlockManagerInfo: Added broadcast_1053_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:56 INFO SparkContext: Created broadcast 1053 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:56 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 696 (MapPartitionsRDD[2123] at repartition at SparkUtils.java:190)
16/09/07 19:10:56 INFO TaskSchedulerImpl: Adding task set 696.0 with 8 tasks
16/09/07 19:10:56 INFO TaskSetManager: Starting task 0.0 in stage 696.0 (TID 5568, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 1.0 in stage 696.0 (TID 5569, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 2.0 in stage 696.0 (TID 5570, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 3.0 in stage 696.0 (TID 5571, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 4.0 in stage 696.0 (TID 5572, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 5.0 in stage 696.0 (TID 5573, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 6.0 in stage 696.0 (TID 5574, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 7.0 in stage 696.0 (TID 5575, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:56 INFO Executor: Running task 0.0 in stage 696.0 (TID 5568)
16/09/07 19:10:56 INFO Executor: Running task 1.0 in stage 696.0 (TID 5569)
16/09/07 19:10:56 INFO Executor: Running task 4.0 in stage 696.0 (TID 5572)
16/09/07 19:10:56 INFO Executor: Running task 5.0 in stage 696.0 (TID 5573)
16/09/07 19:10:56 INFO Executor: Running task 7.0 in stage 696.0 (TID 5575)
16/09/07 19:10:56 INFO Executor: Running task 2.0 in stage 696.0 (TID 5570)
16/09/07 19:10:56 INFO Executor: Running task 3.0 in stage 696.0 (TID 5571)
16/09/07 19:10:56 INFO Executor: Running task 6.0 in stage 696.0 (TID 5574)
16/09/07 19:10:56 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:56 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:56 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:56 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:56 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:56 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:56 INFO Executor: Finished task 2.0 in stage 696.0 (TID 5570). 2260 bytes result sent to driver
16/09/07 19:10:56 INFO Executor: Finished task 7.0 in stage 696.0 (TID 5575). 2260 bytes result sent to driver
16/09/07 19:10:56 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:56 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:56 INFO Executor: Finished task 6.0 in stage 696.0 (TID 5574). 2260 bytes result sent to driver
16/09/07 19:10:56 INFO Executor: Finished task 5.0 in stage 696.0 (TID 5573). 2260 bytes result sent to driver
16/09/07 19:10:56 INFO TaskSetManager: Finished task 2.0 in stage 696.0 (TID 5570) in 3 ms on localhost (1/8)
16/09/07 19:10:56 INFO Executor: Finished task 4.0 in stage 696.0 (TID 5572). 2260 bytes result sent to driver
16/09/07 19:10:56 INFO TaskSetManager: Finished task 7.0 in stage 696.0 (TID 5575) in 3 ms on localhost (2/8)
16/09/07 19:10:56 INFO Executor: Finished task 3.0 in stage 696.0 (TID 5571). 2260 bytes result sent to driver
16/09/07 19:10:56 INFO TaskSetManager: Finished task 6.0 in stage 696.0 (TID 5574) in 3 ms on localhost (3/8)
16/09/07 19:10:56 INFO TaskSetManager: Finished task 5.0 in stage 696.0 (TID 5573) in 3 ms on localhost (4/8)
16/09/07 19:10:56 INFO Executor: Finished task 1.0 in stage 696.0 (TID 5569). 2260 bytes result sent to driver
16/09/07 19:10:56 INFO Executor: Finished task 0.0 in stage 696.0 (TID 5568). 2260 bytes result sent to driver
16/09/07 19:10:56 INFO TaskSetManager: Finished task 4.0 in stage 696.0 (TID 5572) in 3 ms on localhost (5/8)
16/09/07 19:10:56 INFO TaskSetManager: Finished task 3.0 in stage 696.0 (TID 5571) in 3 ms on localhost (6/8)
16/09/07 19:10:56 INFO TaskSetManager: Finished task 1.0 in stage 696.0 (TID 5569) in 3 ms on localhost (7/8)
16/09/07 19:10:56 INFO TaskSetManager: Finished task 0.0 in stage 696.0 (TID 5568) in 3 ms on localhost (8/8)
16/09/07 19:10:56 INFO TaskSchedulerImpl: Removed TaskSet 696.0, whose tasks have all completed, from pool 
16/09/07 19:10:56 INFO DAGScheduler: ShuffleMapStage 696 (repartition at SparkUtils.java:190) finished in 0.003 s
16/09/07 19:10:56 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:56 INFO DAGScheduler: running: Set()
16/09/07 19:10:56 INFO DAGScheduler: waiting: Set(ResultStage 697)
16/09/07 19:10:56 INFO DAGScheduler: failed: Set()
16/09/07 19:10:56 INFO DAGScheduler: Missing parents for ResultStage 697: List()
16/09/07 19:10:56 INFO DAGScheduler: Submitting ResultStage 697 (MapPartitionsRDD[2127] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:56 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19126979, maxMem=4042889625
16/09/07 19:10:56 INFO MemoryStore: Block broadcast_1054 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:56 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19132075, maxMem=4042889625
16/09/07 19:10:56 INFO MemoryStore: Block broadcast_1054_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:56 INFO BlockManagerInfo: Added broadcast_1054_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:56 INFO SparkContext: Created broadcast 1054 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:56 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 697 (MapPartitionsRDD[2127] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:56 INFO TaskSchedulerImpl: Adding task set 697.0 with 8 tasks
16/09/07 19:10:56 INFO TaskSetManager: Starting task 0.0 in stage 697.0 (TID 5576, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 1.0 in stage 697.0 (TID 5577, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 2.0 in stage 697.0 (TID 5578, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 3.0 in stage 697.0 (TID 5579, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 4.0 in stage 697.0 (TID 5580, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 5.0 in stage 697.0 (TID 5581, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 6.0 in stage 697.0 (TID 5582, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:56 INFO TaskSetManager: Starting task 7.0 in stage 697.0 (TID 5583, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:56 INFO Executor: Running task 0.0 in stage 697.0 (TID 5576)
16/09/07 19:10:56 INFO Executor: Running task 3.0 in stage 697.0 (TID 5579)
16/09/07 19:10:56 INFO Executor: Running task 7.0 in stage 697.0 (TID 5583)
16/09/07 19:10:56 INFO Executor: Running task 2.0 in stage 697.0 (TID 5578)
16/09/07 19:10:56 INFO Executor: Running task 4.0 in stage 697.0 (TID 5580)
16/09/07 19:10:56 INFO Executor: Running task 6.0 in stage 697.0 (TID 5582)
16/09/07 19:10:56 INFO Executor: Running task 5.0 in stage 697.0 (TID 5581)
16/09/07 19:10:56 INFO Executor: Running task 1.0 in stage 697.0 (TID 5577)
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:56 INFO Executor: Finished task 1.0 in stage 697.0 (TID 5577). 2160 bytes result sent to driver
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:56 INFO Executor: Finished task 4.0 in stage 697.0 (TID 5580). 2160 bytes result sent to driver
16/09/07 19:10:56 INFO TaskSetManager: Finished task 1.0 in stage 697.0 (TID 5577) in 2 ms on localhost (1/8)
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:56 INFO TaskSetManager: Finished task 4.0 in stage 697.0 (TID 5580) in 2 ms on localhost (2/8)
16/09/07 19:10:56 INFO Executor: Finished task 5.0 in stage 697.0 (TID 5581). 2160 bytes result sent to driver
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:56 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:56 INFO TaskSetManager: Finished task 5.0 in stage 697.0 (TID 5581) in 2 ms on localhost (3/8)
16/09/07 19:10:56 INFO Executor: Finished task 2.0 in stage 697.0 (TID 5578). 2160 bytes result sent to driver
16/09/07 19:10:56 INFO Executor: Finished task 3.0 in stage 697.0 (TID 5579). 2160 bytes result sent to driver
16/09/07 19:10:56 INFO TaskSetManager: Finished task 2.0 in stage 697.0 (TID 5578) in 3 ms on localhost (4/8)
16/09/07 19:10:56 INFO TaskSetManager: Finished task 3.0 in stage 697.0 (TID 5579) in 3 ms on localhost (5/8)
16/09/07 19:10:56 INFO BlockManagerInfo: Removed broadcast_1053_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:57 INFO Executor: Finished task 6.0 in stage 697.0 (TID 5582). 3485274 bytes result sent to driver
16/09/07 19:10:57 INFO Executor: Finished task 0.0 in stage 697.0 (TID 5576). 3485274 bytes result sent to driver
16/09/07 19:10:57 INFO Executor: Finished task 7.0 in stage 697.0 (TID 5583). 3485274 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 7.0 in stage 697.0 (TID 5583) in 914 ms on localhost (6/8)
16/09/07 19:10:57 INFO TaskSetManager: Finished task 6.0 in stage 697.0 (TID 5582) in 915 ms on localhost (7/8)
16/09/07 19:10:57 INFO TaskSetManager: Finished task 0.0 in stage 697.0 (TID 5576) in 915 ms on localhost (8/8)
16/09/07 19:10:57 INFO TaskSchedulerImpl: Removed TaskSet 697.0, whose tasks have all completed, from pool 
16/09/07 19:10:57 INFO DAGScheduler: ResultStage 697 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.915 s
16/09/07 19:10:57 INFO DAGScheduler: Job 353 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.922505 s
16/09/07 19:10:57 INFO ParameterAveragingTrainingMaster: Completed training of split 32 of 78
16/09/07 19:10:57 INFO ParameterAveragingTrainingMaster: Starting training of split 33 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:57 INFO MemoryStore: ensureFreeSpace(18425296) called with curMem=19127792, maxMem=4042889625
16/09/07 19:10:57 INFO MemoryStore: Block broadcast_1055 stored as values in memory (estimated size 17.6 MB, free 3.7 GB)
16/09/07 19:10:57 INFO MemoryStore: ensureFreeSpace(168226) called with curMem=37553088, maxMem=4042889625
16/09/07 19:10:57 INFO MemoryStore: Block broadcast_1055_piece0 stored as bytes in memory (estimated size 164.3 KB, free 3.7 GB)
16/09/07 19:10:57 INFO BlockManagerInfo: Added broadcast_1055_piece0 in memory on localhost:41251 (size: 164.3 KB, free: 3.8 GB)
16/09/07 19:10:57 INFO SparkContext: Created broadcast 1055 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:57 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:57 INFO DAGScheduler: Registering RDD 2128 (repartition at SparkUtils.java:190)
16/09/07 19:10:57 INFO DAGScheduler: Got job 354 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:57 INFO DAGScheduler: Final stage: ResultStage 699(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:57 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 698)
16/09/07 19:10:57 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 698)
16/09/07 19:10:57 INFO DAGScheduler: Submitting ShuffleMapStage 698 (MapPartitionsRDD[2128] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:57 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=37721314, maxMem=4042889625
16/09/07 19:10:57 INFO MemoryStore: Block broadcast_1056 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:57 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=37725778, maxMem=4042889625
16/09/07 19:10:57 INFO MemoryStore: Block broadcast_1056_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:57 INFO BlockManagerInfo: Added broadcast_1056_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:57 INFO SparkContext: Created broadcast 1056 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:57 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 698 (MapPartitionsRDD[2128] at repartition at SparkUtils.java:190)
16/09/07 19:10:57 INFO TaskSchedulerImpl: Adding task set 698.0 with 8 tasks
16/09/07 19:10:57 INFO TaskSetManager: Starting task 0.0 in stage 698.0 (TID 5584, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 1.0 in stage 698.0 (TID 5585, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 2.0 in stage 698.0 (TID 5586, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 3.0 in stage 698.0 (TID 5587, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 4.0 in stage 698.0 (TID 5588, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 5.0 in stage 698.0 (TID 5589, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 6.0 in stage 698.0 (TID 5590, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 7.0 in stage 698.0 (TID 5591, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:57 INFO Executor: Running task 3.0 in stage 698.0 (TID 5587)
16/09/07 19:10:57 INFO Executor: Running task 4.0 in stage 698.0 (TID 5588)
16/09/07 19:10:57 INFO Executor: Running task 2.0 in stage 698.0 (TID 5586)
16/09/07 19:10:57 INFO Executor: Running task 1.0 in stage 698.0 (TID 5585)
16/09/07 19:10:57 INFO Executor: Running task 5.0 in stage 698.0 (TID 5589)
16/09/07 19:10:57 INFO Executor: Running task 6.0 in stage 698.0 (TID 5590)
16/09/07 19:10:57 INFO Executor: Running task 0.0 in stage 698.0 (TID 5584)
16/09/07 19:10:57 INFO Executor: Running task 7.0 in stage 698.0 (TID 5591)
16/09/07 19:10:57 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:57 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:57 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:57 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:57 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:57 INFO Executor: Finished task 3.0 in stage 698.0 (TID 5587). 2260 bytes result sent to driver
16/09/07 19:10:57 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:57 INFO Executor: Finished task 5.0 in stage 698.0 (TID 5589). 2260 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 3.0 in stage 698.0 (TID 5587) in 2 ms on localhost (1/8)
16/09/07 19:10:57 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:57 INFO TaskSetManager: Finished task 5.0 in stage 698.0 (TID 5589) in 2 ms on localhost (2/8)
16/09/07 19:10:57 INFO Executor: Finished task 2.0 in stage 698.0 (TID 5586). 2260 bytes result sent to driver
16/09/07 19:10:57 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:57 INFO TaskSetManager: Finished task 2.0 in stage 698.0 (TID 5586) in 2 ms on localhost (3/8)
16/09/07 19:10:57 INFO Executor: Finished task 1.0 in stage 698.0 (TID 5585). 2260 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 1.0 in stage 698.0 (TID 5585) in 3 ms on localhost (4/8)
16/09/07 19:10:57 INFO Executor: Finished task 4.0 in stage 698.0 (TID 5588). 2260 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 4.0 in stage 698.0 (TID 5588) in 4 ms on localhost (5/8)
16/09/07 19:10:57 INFO Executor: Finished task 6.0 in stage 698.0 (TID 5590). 2260 bytes result sent to driver
16/09/07 19:10:57 INFO Executor: Finished task 7.0 in stage 698.0 (TID 5591). 2260 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 7.0 in stage 698.0 (TID 5591) in 5 ms on localhost (6/8)
16/09/07 19:10:57 INFO TaskSetManager: Finished task 6.0 in stage 698.0 (TID 5590) in 5 ms on localhost (7/8)
16/09/07 19:10:57 INFO Executor: Finished task 0.0 in stage 698.0 (TID 5584). 2260 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 0.0 in stage 698.0 (TID 5584) in 5 ms on localhost (8/8)
16/09/07 19:10:57 INFO TaskSchedulerImpl: Removed TaskSet 698.0, whose tasks have all completed, from pool 
16/09/07 19:10:57 INFO DAGScheduler: ShuffleMapStage 698 (repartition at SparkUtils.java:190) finished in 0.076 s
16/09/07 19:10:57 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:57 INFO DAGScheduler: running: Set()
16/09/07 19:10:57 INFO DAGScheduler: waiting: Set(ResultStage 699)
16/09/07 19:10:57 INFO DAGScheduler: failed: Set()
16/09/07 19:10:57 INFO DAGScheduler: Missing parents for ResultStage 699: List()
16/09/07 19:10:57 INFO BlockManagerInfo: Removed broadcast_1054_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:57 INFO DAGScheduler: Submitting ResultStage 699 (MapPartitionsRDD[2132] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:57 INFO ContextCleaner: Cleaned accumulator 698
16/09/07 19:10:57 INFO ContextCleaner: Cleaned accumulator 697
16/09/07 19:10:57 INFO BlockManagerInfo: Removed broadcast_1052_piece0 on localhost:41251 in memory (size: 164.7 KB, free: 3.8 GB)
16/09/07 19:10:57 INFO ContextCleaner: Cleaned shuffle 343
16/09/07 19:10:57 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19144374, maxMem=4042889625
16/09/07 19:10:57 INFO MemoryStore: Block broadcast_1057 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:57 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19149470, maxMem=4042889625
16/09/07 19:10:57 INFO MemoryStore: Block broadcast_1057_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:57 INFO BlockManagerInfo: Added broadcast_1057_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:57 INFO SparkContext: Created broadcast 1057 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:57 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 699 (MapPartitionsRDD[2132] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:57 INFO TaskSchedulerImpl: Adding task set 699.0 with 8 tasks
16/09/07 19:10:57 INFO TaskSetManager: Starting task 0.0 in stage 699.0 (TID 5592, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 1.0 in stage 699.0 (TID 5593, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 2.0 in stage 699.0 (TID 5594, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 3.0 in stage 699.0 (TID 5595, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 4.0 in stage 699.0 (TID 5596, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 5.0 in stage 699.0 (TID 5597, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 6.0 in stage 699.0 (TID 5598, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:57 INFO TaskSetManager: Starting task 7.0 in stage 699.0 (TID 5599, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:57 INFO Executor: Running task 0.0 in stage 699.0 (TID 5592)
16/09/07 19:10:57 INFO Executor: Running task 6.0 in stage 699.0 (TID 5598)
16/09/07 19:10:57 INFO Executor: Running task 1.0 in stage 699.0 (TID 5593)
16/09/07 19:10:57 INFO Executor: Running task 2.0 in stage 699.0 (TID 5594)
16/09/07 19:10:57 INFO Executor: Running task 3.0 in stage 699.0 (TID 5595)
16/09/07 19:10:57 INFO Executor: Running task 4.0 in stage 699.0 (TID 5596)
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:57 INFO Executor: Running task 5.0 in stage 699.0 (TID 5597)
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:57 INFO Executor: Finished task 4.0 in stage 699.0 (TID 5596). 2160 bytes result sent to driver
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:57 INFO TaskSetManager: Finished task 4.0 in stage 699.0 (TID 5596) in 2 ms on localhost (1/8)
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:57 INFO Executor: Finished task 5.0 in stage 699.0 (TID 5597). 2160 bytes result sent to driver
16/09/07 19:10:57 INFO Executor: Running task 7.0 in stage 699.0 (TID 5599)
16/09/07 19:10:57 INFO TaskSetManager: Finished task 5.0 in stage 699.0 (TID 5597) in 3 ms on localhost (2/8)
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:57 INFO Executor: Finished task 3.0 in stage 699.0 (TID 5595). 2160 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 3.0 in stage 699.0 (TID 5595) in 4 ms on localhost (3/8)
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:57 INFO Executor: Finished task 2.0 in stage 699.0 (TID 5594). 2160 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 2.0 in stage 699.0 (TID 5594) in 5 ms on localhost (4/8)
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:57 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:57 INFO Executor: Finished task 1.0 in stage 699.0 (TID 5593). 2160 bytes result sent to driver
16/09/07 19:10:57 INFO TaskSetManager: Finished task 1.0 in stage 699.0 (TID 5593) in 6 ms on localhost (5/8)
16/09/07 19:10:57 INFO BlockManagerInfo: Removed broadcast_1056_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:58 INFO Executor: Finished task 7.0 in stage 699.0 (TID 5599). 3485274 bytes result sent to driver
16/09/07 19:10:58 INFO Executor: Finished task 0.0 in stage 699.0 (TID 5592). 3485274 bytes result sent to driver
16/09/07 19:10:58 INFO TaskSetManager: Finished task 7.0 in stage 699.0 (TID 5599) in 961 ms on localhost (6/8)
16/09/07 19:10:58 INFO TaskSetManager: Finished task 0.0 in stage 699.0 (TID 5592) in 977 ms on localhost (7/8)
16/09/07 19:10:58 INFO Executor: Finished task 6.0 in stage 699.0 (TID 5598). 3485274 bytes result sent to driver
16/09/07 19:10:58 INFO TaskSetManager: Finished task 6.0 in stage 699.0 (TID 5598) in 1008 ms on localhost (8/8)
16/09/07 19:10:58 INFO DAGScheduler: ResultStage 699 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 1.008 s
16/09/07 19:10:58 INFO TaskSchedulerImpl: Removed TaskSet 699.0, whose tasks have all completed, from pool 
16/09/07 19:10:58 INFO DAGScheduler: Job 354 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.088916 s
16/09/07 19:10:58 INFO ParameterAveragingTrainingMaster: Completed training of split 33 of 78
16/09/07 19:10:58 INFO ParameterAveragingTrainingMaster: Starting training of split 34 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:10:58 INFO MemoryStore: ensureFreeSpace(18428968) called with curMem=19145185, maxMem=4042889625
16/09/07 19:10:58 INFO MemoryStore: Block broadcast_1058 stored as values in memory (estimated size 17.6 MB, free 3.7 GB)
16/09/07 19:10:58 INFO BlockManagerInfo: Removed broadcast_1057_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:58 INFO ContextCleaner: Cleaned accumulator 700
16/09/07 19:10:58 INFO ContextCleaner: Cleaned accumulator 699
16/09/07 19:10:58 INFO BlockManagerInfo: Removed broadcast_1055_piece0 on localhost:41251 in memory (size: 164.3 KB, free: 3.8 GB)
16/09/07 19:10:58 INFO ContextCleaner: Cleaned shuffle 344
16/09/07 19:10:58 INFO MemoryStore: ensureFreeSpace(168160) called with curMem=18972792, maxMem=4042889625
16/09/07 19:10:58 INFO MemoryStore: Block broadcast_1058_piece0 stored as bytes in memory (estimated size 164.2 KB, free 3.7 GB)
16/09/07 19:10:58 INFO BlockManagerInfo: Added broadcast_1058_piece0 in memory on localhost:41251 (size: 164.2 KB, free: 3.8 GB)
16/09/07 19:10:58 INFO SparkContext: Created broadcast 1058 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:10:58 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:10:58 INFO DAGScheduler: Registering RDD 2133 (repartition at SparkUtils.java:190)
16/09/07 19:10:58 INFO DAGScheduler: Got job 355 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:10:58 INFO DAGScheduler: Final stage: ResultStage 701(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:10:58 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 700)
16/09/07 19:10:58 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 700)
16/09/07 19:10:58 INFO DAGScheduler: Submitting ShuffleMapStage 700 (MapPartitionsRDD[2133] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:10:58 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19140952, maxMem=4042889625
16/09/07 19:10:58 INFO MemoryStore: Block broadcast_1059 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:10:58 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19145416, maxMem=4042889625
16/09/07 19:10:58 INFO MemoryStore: Block broadcast_1059_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:10:58 INFO BlockManagerInfo: Added broadcast_1059_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:58 INFO SparkContext: Created broadcast 1059 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:58 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 700 (MapPartitionsRDD[2133] at repartition at SparkUtils.java:190)
16/09/07 19:10:58 INFO TaskSchedulerImpl: Adding task set 700.0 with 8 tasks
16/09/07 19:10:58 INFO TaskSetManager: Starting task 0.0 in stage 700.0 (TID 5600, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 1.0 in stage 700.0 (TID 5601, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 2.0 in stage 700.0 (TID 5602, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 3.0 in stage 700.0 (TID 5603, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 4.0 in stage 700.0 (TID 5604, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 5.0 in stage 700.0 (TID 5605, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 6.0 in stage 700.0 (TID 5606, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 7.0 in stage 700.0 (TID 5607, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:10:58 INFO Executor: Running task 1.0 in stage 700.0 (TID 5601)
16/09/07 19:10:58 INFO Executor: Running task 3.0 in stage 700.0 (TID 5603)
16/09/07 19:10:58 INFO Executor: Running task 7.0 in stage 700.0 (TID 5607)
16/09/07 19:10:58 INFO Executor: Running task 4.0 in stage 700.0 (TID 5604)
16/09/07 19:10:58 INFO Executor: Running task 5.0 in stage 700.0 (TID 5605)
16/09/07 19:10:58 INFO Executor: Running task 0.0 in stage 700.0 (TID 5600)
16/09/07 19:10:58 INFO Executor: Running task 2.0 in stage 700.0 (TID 5602)
16/09/07 19:10:58 INFO Executor: Running task 6.0 in stage 700.0 (TID 5606)
16/09/07 19:10:58 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:10:58 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:10:58 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:10:58 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:10:58 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:10:58 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:10:58 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:10:58 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:10:58 INFO Executor: Finished task 1.0 in stage 700.0 (TID 5601). 2260 bytes result sent to driver
16/09/07 19:10:58 INFO Executor: Finished task 0.0 in stage 700.0 (TID 5600). 2260 bytes result sent to driver
16/09/07 19:10:58 INFO Executor: Finished task 3.0 in stage 700.0 (TID 5603). 2260 bytes result sent to driver
16/09/07 19:10:58 INFO Executor: Finished task 4.0 in stage 700.0 (TID 5604). 2260 bytes result sent to driver
16/09/07 19:10:58 INFO TaskSetManager: Finished task 0.0 in stage 700.0 (TID 5600) in 3 ms on localhost (1/8)
16/09/07 19:10:58 INFO TaskSetManager: Finished task 3.0 in stage 700.0 (TID 5603) in 3 ms on localhost (2/8)
16/09/07 19:10:58 INFO TaskSetManager: Finished task 4.0 in stage 700.0 (TID 5604) in 3 ms on localhost (3/8)
16/09/07 19:10:58 INFO TaskSetManager: Finished task 1.0 in stage 700.0 (TID 5601) in 3 ms on localhost (4/8)
16/09/07 19:10:58 INFO Executor: Finished task 5.0 in stage 700.0 (TID 5605). 2260 bytes result sent to driver
16/09/07 19:10:58 INFO TaskSetManager: Finished task 5.0 in stage 700.0 (TID 5605) in 3 ms on localhost (5/8)
16/09/07 19:10:58 INFO Executor: Finished task 6.0 in stage 700.0 (TID 5606). 2260 bytes result sent to driver
16/09/07 19:10:58 INFO Executor: Finished task 2.0 in stage 700.0 (TID 5602). 2260 bytes result sent to driver
16/09/07 19:10:58 INFO TaskSetManager: Finished task 6.0 in stage 700.0 (TID 5606) in 3 ms on localhost (6/8)
16/09/07 19:10:58 INFO Executor: Finished task 7.0 in stage 700.0 (TID 5607). 2260 bytes result sent to driver
16/09/07 19:10:58 INFO TaskSetManager: Finished task 2.0 in stage 700.0 (TID 5602) in 4 ms on localhost (7/8)
16/09/07 19:10:58 INFO TaskSetManager: Finished task 7.0 in stage 700.0 (TID 5607) in 4 ms on localhost (8/8)
16/09/07 19:10:58 INFO TaskSchedulerImpl: Removed TaskSet 700.0, whose tasks have all completed, from pool 
16/09/07 19:10:58 INFO DAGScheduler: ShuffleMapStage 700 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:10:58 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:10:58 INFO DAGScheduler: running: Set()
16/09/07 19:10:58 INFO DAGScheduler: waiting: Set(ResultStage 701)
16/09/07 19:10:58 INFO DAGScheduler: failed: Set()
16/09/07 19:10:58 INFO DAGScheduler: Missing parents for ResultStage 701: List()
16/09/07 19:10:58 INFO DAGScheduler: Submitting ResultStage 701 (MapPartitionsRDD[2137] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:10:58 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19147980, maxMem=4042889625
16/09/07 19:10:58 INFO MemoryStore: Block broadcast_1060 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:10:58 INFO MemoryStore: ensureFreeSpace(2745) called with curMem=19153076, maxMem=4042889625
16/09/07 19:10:58 INFO MemoryStore: Block broadcast_1060_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:10:58 INFO BlockManagerInfo: Added broadcast_1060_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:10:58 INFO SparkContext: Created broadcast 1060 from broadcast at DAGScheduler.scala:861
16/09/07 19:10:58 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 701 (MapPartitionsRDD[2137] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:10:58 INFO TaskSchedulerImpl: Adding task set 701.0 with 8 tasks
16/09/07 19:10:58 INFO TaskSetManager: Starting task 0.0 in stage 701.0 (TID 5608, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 1.0 in stage 701.0 (TID 5609, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 2.0 in stage 701.0 (TID 5610, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 3.0 in stage 701.0 (TID 5611, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 4.0 in stage 701.0 (TID 5612, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 5.0 in stage 701.0 (TID 5613, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 6.0 in stage 701.0 (TID 5614, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:58 INFO TaskSetManager: Starting task 7.0 in stage 701.0 (TID 5615, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:10:58 INFO Executor: Running task 0.0 in stage 701.0 (TID 5608)
16/09/07 19:10:58 INFO Executor: Running task 5.0 in stage 701.0 (TID 5613)
16/09/07 19:10:58 INFO Executor: Running task 7.0 in stage 701.0 (TID 5615)
16/09/07 19:10:58 INFO Executor: Running task 1.0 in stage 701.0 (TID 5609)
16/09/07 19:10:58 INFO Executor: Running task 2.0 in stage 701.0 (TID 5610)
16/09/07 19:10:58 INFO Executor: Running task 6.0 in stage 701.0 (TID 5614)
16/09/07 19:10:58 INFO Executor: Running task 4.0 in stage 701.0 (TID 5612)
16/09/07 19:10:58 INFO Executor: Running task 3.0 in stage 701.0 (TID 5611)
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:58 INFO Executor: Finished task 3.0 in stage 701.0 (TID 5611). 2160 bytes result sent to driver
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:58 INFO Executor: Finished task 4.0 in stage 701.0 (TID 5612). 2160 bytes result sent to driver
16/09/07 19:10:58 INFO Executor: Finished task 5.0 in stage 701.0 (TID 5613). 2160 bytes result sent to driver
16/09/07 19:10:58 INFO TaskSetManager: Finished task 3.0 in stage 701.0 (TID 5611) in 2 ms on localhost (1/8)
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:58 INFO TaskSetManager: Finished task 4.0 in stage 701.0 (TID 5612) in 2 ms on localhost (2/8)
16/09/07 19:10:58 INFO TaskSetManager: Finished task 5.0 in stage 701.0 (TID 5613) in 2 ms on localhost (3/8)
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:58 INFO Executor: Finished task 2.0 in stage 701.0 (TID 5610). 2160 bytes result sent to driver
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:10:58 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:10:58 INFO TaskSetManager: Finished task 2.0 in stage 701.0 (TID 5610) in 3 ms on localhost (4/8)
16/09/07 19:10:58 INFO Executor: Finished task 1.0 in stage 701.0 (TID 5609). 2160 bytes result sent to driver
16/09/07 19:10:58 INFO TaskSetManager: Finished task 1.0 in stage 701.0 (TID 5609) in 4 ms on localhost (5/8)
16/09/07 19:10:59 INFO BlockManagerInfo: Removed broadcast_1059_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:10:59 INFO Executor: Finished task 6.0 in stage 701.0 (TID 5614). 3485274 bytes result sent to driver
16/09/07 19:10:59 INFO TaskSetManager: Finished task 6.0 in stage 701.0 (TID 5614) in 920 ms on localhost (6/8)
16/09/07 19:10:59 INFO Executor: Finished task 7.0 in stage 701.0 (TID 5615). 3485274 bytes result sent to driver
16/09/07 19:10:59 INFO Executor: Finished task 0.0 in stage 701.0 (TID 5608). 3485274 bytes result sent to driver
16/09/07 19:10:59 INFO TaskSetManager: Finished task 7.0 in stage 701.0 (TID 5615) in 1093 ms on localhost (7/8)
16/09/07 19:10:59 INFO TaskSetManager: Finished task 0.0 in stage 701.0 (TID 5608) in 1093 ms on localhost (8/8)
16/09/07 19:10:59 INFO TaskSchedulerImpl: Removed TaskSet 701.0, whose tasks have all completed, from pool 
16/09/07 19:10:59 INFO DAGScheduler: ResultStage 701 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 1.093 s
16/09/07 19:10:59 INFO DAGScheduler: Job 355 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.101031 s
16/09/07 19:10:59 INFO ParameterAveragingTrainingMaster: Completed training of split 34 of 78
16/09/07 19:10:59 INFO ParameterAveragingTrainingMaster: Starting training of split 35 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:00 INFO MemoryStore: ensureFreeSpace(18435344) called with curMem=19148793, maxMem=4042889625
16/09/07 19:11:00 INFO MemoryStore: Block broadcast_1061 stored as values in memory (estimated size 17.6 MB, free 3.7 GB)
16/09/07 19:11:00 INFO MemoryStore: ensureFreeSpace(168070) called with curMem=37584137, maxMem=4042889625
16/09/07 19:11:00 INFO MemoryStore: Block broadcast_1061_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:00 INFO BlockManagerInfo: Added broadcast_1061_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:00 INFO SparkContext: Created broadcast 1061 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:00 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:00 INFO DAGScheduler: Registering RDD 2138 (repartition at SparkUtils.java:190)
16/09/07 19:11:00 INFO DAGScheduler: Got job 356 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:00 INFO DAGScheduler: Final stage: ResultStage 703(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:00 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 702)
16/09/07 19:11:00 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 702)
16/09/07 19:11:00 INFO DAGScheduler: Submitting ShuffleMapStage 702 (MapPartitionsRDD[2138] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:00 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=37752207, maxMem=4042889625
16/09/07 19:11:00 INFO MemoryStore: Block broadcast_1062 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:00 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=37756671, maxMem=4042889625
16/09/07 19:11:00 INFO MemoryStore: Block broadcast_1062_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:00 INFO BlockManagerInfo: Added broadcast_1062_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:00 INFO SparkContext: Created broadcast 1062 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:00 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 702 (MapPartitionsRDD[2138] at repartition at SparkUtils.java:190)
16/09/07 19:11:00 INFO TaskSchedulerImpl: Adding task set 702.0 with 8 tasks
16/09/07 19:11:00 INFO TaskSetManager: Starting task 0.0 in stage 702.0 (TID 5616, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 1.0 in stage 702.0 (TID 5617, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 2.0 in stage 702.0 (TID 5618, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 3.0 in stage 702.0 (TID 5619, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 4.0 in stage 702.0 (TID 5620, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 5.0 in stage 702.0 (TID 5621, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 6.0 in stage 702.0 (TID 5622, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 7.0 in stage 702.0 (TID 5623, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:00 INFO Executor: Running task 0.0 in stage 702.0 (TID 5616)
16/09/07 19:11:00 INFO Executor: Running task 2.0 in stage 702.0 (TID 5618)
16/09/07 19:11:00 INFO Executor: Running task 3.0 in stage 702.0 (TID 5619)
16/09/07 19:11:00 INFO Executor: Running task 6.0 in stage 702.0 (TID 5622)
16/09/07 19:11:00 INFO Executor: Running task 5.0 in stage 702.0 (TID 5621)
16/09/07 19:11:00 INFO Executor: Running task 4.0 in stage 702.0 (TID 5620)
16/09/07 19:11:00 INFO Executor: Running task 1.0 in stage 702.0 (TID 5617)
16/09/07 19:11:00 INFO Executor: Running task 7.0 in stage 702.0 (TID 5623)
16/09/07 19:11:00 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:00 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:00 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:00 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:00 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:00 INFO Executor: Finished task 3.0 in stage 702.0 (TID 5619). 2260 bytes result sent to driver
16/09/07 19:11:00 INFO TaskSetManager: Finished task 3.0 in stage 702.0 (TID 5619) in 2 ms on localhost (1/8)
16/09/07 19:11:00 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:00 INFO Executor: Finished task 7.0 in stage 702.0 (TID 5623). 2260 bytes result sent to driver
16/09/07 19:11:00 INFO Executor: Finished task 1.0 in stage 702.0 (TID 5617). 2260 bytes result sent to driver
16/09/07 19:11:00 INFO TaskSetManager: Finished task 7.0 in stage 702.0 (TID 5623) in 2 ms on localhost (2/8)
16/09/07 19:11:00 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:00 INFO TaskSetManager: Finished task 1.0 in stage 702.0 (TID 5617) in 3 ms on localhost (3/8)
16/09/07 19:11:00 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:00 INFO Executor: Finished task 0.0 in stage 702.0 (TID 5616). 2260 bytes result sent to driver
16/09/07 19:11:00 INFO Executor: Finished task 2.0 in stage 702.0 (TID 5618). 2260 bytes result sent to driver
16/09/07 19:11:00 INFO Executor: Finished task 5.0 in stage 702.0 (TID 5621). 2260 bytes result sent to driver
16/09/07 19:11:00 INFO Executor: Finished task 6.0 in stage 702.0 (TID 5622). 2260 bytes result sent to driver
16/09/07 19:11:00 INFO TaskSetManager: Finished task 0.0 in stage 702.0 (TID 5616) in 4 ms on localhost (4/8)
16/09/07 19:11:00 INFO TaskSetManager: Finished task 2.0 in stage 702.0 (TID 5618) in 4 ms on localhost (5/8)
16/09/07 19:11:00 INFO TaskSetManager: Finished task 5.0 in stage 702.0 (TID 5621) in 4 ms on localhost (6/8)
16/09/07 19:11:00 INFO TaskSetManager: Finished task 6.0 in stage 702.0 (TID 5622) in 4 ms on localhost (7/8)
16/09/07 19:11:00 INFO Executor: Finished task 4.0 in stage 702.0 (TID 5620). 2260 bytes result sent to driver
16/09/07 19:11:00 INFO TaskSetManager: Finished task 4.0 in stage 702.0 (TID 5620) in 4 ms on localhost (8/8)
16/09/07 19:11:00 INFO TaskSchedulerImpl: Removed TaskSet 702.0, whose tasks have all completed, from pool 
16/09/07 19:11:00 INFO DAGScheduler: ShuffleMapStage 702 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:00 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:00 INFO DAGScheduler: running: Set()
16/09/07 19:11:00 INFO DAGScheduler: waiting: Set(ResultStage 703)
16/09/07 19:11:00 INFO DAGScheduler: failed: Set()
16/09/07 19:11:00 INFO DAGScheduler: Missing parents for ResultStage 703: List()
16/09/07 19:11:00 INFO DAGScheduler: Submitting ResultStage 703 (MapPartitionsRDD[2142] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:00 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=37759235, maxMem=4042889625
16/09/07 19:11:00 INFO MemoryStore: Block broadcast_1063 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:00 INFO MemoryStore: ensureFreeSpace(2742) called with curMem=37764331, maxMem=4042889625
16/09/07 19:11:00 INFO MemoryStore: Block broadcast_1063_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:00 INFO BlockManagerInfo: Added broadcast_1063_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:00 INFO SparkContext: Created broadcast 1063 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:00 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 703 (MapPartitionsRDD[2142] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:00 INFO TaskSchedulerImpl: Adding task set 703.0 with 8 tasks
16/09/07 19:11:00 INFO TaskSetManager: Starting task 0.0 in stage 703.0 (TID 5624, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 1.0 in stage 703.0 (TID 5625, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 2.0 in stage 703.0 (TID 5626, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 3.0 in stage 703.0 (TID 5627, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 4.0 in stage 703.0 (TID 5628, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 5.0 in stage 703.0 (TID 5629, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 6.0 in stage 703.0 (TID 5630, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:00 INFO TaskSetManager: Starting task 7.0 in stage 703.0 (TID 5631, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:00 INFO Executor: Running task 2.0 in stage 703.0 (TID 5626)
16/09/07 19:11:00 INFO Executor: Running task 3.0 in stage 703.0 (TID 5627)
16/09/07 19:11:00 INFO Executor: Running task 6.0 in stage 703.0 (TID 5630)
16/09/07 19:11:00 INFO Executor: Running task 5.0 in stage 703.0 (TID 5629)
16/09/07 19:11:00 INFO Executor: Running task 0.0 in stage 703.0 (TID 5624)
16/09/07 19:11:00 INFO Executor: Running task 7.0 in stage 703.0 (TID 5631)
16/09/07 19:11:00 INFO Executor: Running task 1.0 in stage 703.0 (TID 5625)
16/09/07 19:11:00 INFO Executor: Running task 4.0 in stage 703.0 (TID 5628)
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:00 INFO Executor: Finished task 2.0 in stage 703.0 (TID 5626). 2160 bytes result sent to driver
16/09/07 19:11:00 INFO Executor: Finished task 5.0 in stage 703.0 (TID 5629). 2160 bytes result sent to driver
16/09/07 19:11:00 INFO TaskSetManager: Finished task 2.0 in stage 703.0 (TID 5626) in 2 ms on localhost (1/8)
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:00 INFO TaskSetManager: Finished task 5.0 in stage 703.0 (TID 5629) in 2 ms on localhost (2/8)
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:00 INFO Executor: Finished task 3.0 in stage 703.0 (TID 5627). 2160 bytes result sent to driver
16/09/07 19:11:00 INFO TaskSetManager: Finished task 3.0 in stage 703.0 (TID 5627) in 3 ms on localhost (3/8)
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:00 INFO Executor: Finished task 4.0 in stage 703.0 (TID 5628). 2160 bytes result sent to driver
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:00 INFO TaskSetManager: Finished task 4.0 in stage 703.0 (TID 5628) in 5 ms on localhost (4/8)
16/09/07 19:11:00 INFO Executor: Finished task 1.0 in stage 703.0 (TID 5625). 2160 bytes result sent to driver
16/09/07 19:11:00 INFO TaskSetManager: Finished task 1.0 in stage 703.0 (TID 5625) in 5 ms on localhost (5/8)
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:00 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:00 INFO BlockManagerInfo: Removed broadcast_1062_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:00 INFO BlockManagerInfo: Removed broadcast_1060_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:00 INFO ContextCleaner: Cleaned accumulator 702
16/09/07 19:11:00 INFO ContextCleaner: Cleaned accumulator 701
16/09/07 19:11:00 INFO BlockManagerInfo: Removed broadcast_1058_piece0 on localhost:41251 in memory (size: 164.2 KB, free: 3.8 GB)
16/09/07 19:11:00 INFO ContextCleaner: Cleaned shuffle 345
16/09/07 19:11:00 INFO Executor: Finished task 6.0 in stage 703.0 (TID 5630). 3485274 bytes result sent to driver
16/09/07 19:11:01 INFO TaskSetManager: Finished task 6.0 in stage 703.0 (TID 5630) in 806 ms on localhost (6/8)
16/09/07 19:11:01 INFO Executor: Finished task 0.0 in stage 703.0 (TID 5624). 3485274 bytes result sent to driver
16/09/07 19:11:01 INFO Executor: Finished task 7.0 in stage 703.0 (TID 5631). 3485274 bytes result sent to driver
16/09/07 19:11:01 INFO TaskSetManager: Finished task 0.0 in stage 703.0 (TID 5624) in 852 ms on localhost (7/8)
16/09/07 19:11:01 INFO TaskSetManager: Finished task 7.0 in stage 703.0 (TID 5631) in 853 ms on localhost (8/8)
16/09/07 19:11:01 INFO TaskSchedulerImpl: Removed TaskSet 703.0, whose tasks have all completed, from pool 
16/09/07 19:11:01 INFO DAGScheduler: ResultStage 703 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.853 s
16/09/07 19:11:01 INFO DAGScheduler: Job 356 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.861499 s
16/09/07 19:11:01 INFO ParameterAveragingTrainingMaster: Completed training of split 35 of 78
16/09/07 19:11:01 INFO ParameterAveragingTrainingMaster: Starting training of split 36 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:01 INFO BlockManagerInfo: Removed broadcast_1063_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:01 INFO ContextCleaner: Cleaned accumulator 704
16/09/07 19:11:01 INFO ContextCleaner: Cleaned accumulator 703
16/09/07 19:11:01 INFO BlockManagerInfo: Removed broadcast_1061_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:01 INFO ContextCleaner: Cleaned shuffle 346
16/09/07 19:11:01 INFO MemoryStore: ensureFreeSpace(18308712) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:01 INFO MemoryStore: Block broadcast_1064 stored as values in memory (estimated size 17.5 MB, free 3.7 GB)
16/09/07 19:11:01 INFO MemoryStore: ensureFreeSpace(168059) called with curMem=18852536, maxMem=4042889625
16/09/07 19:11:01 INFO MemoryStore: Block broadcast_1064_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:01 INFO BlockManagerInfo: Added broadcast_1064_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:01 INFO SparkContext: Created broadcast 1064 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:01 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:01 INFO DAGScheduler: Registering RDD 2143 (repartition at SparkUtils.java:190)
16/09/07 19:11:01 INFO DAGScheduler: Got job 357 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:01 INFO DAGScheduler: Final stage: ResultStage 705(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:01 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 704)
16/09/07 19:11:01 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 704)
16/09/07 19:11:01 INFO DAGScheduler: Submitting ShuffleMapStage 704 (MapPartitionsRDD[2143] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:01 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19020595, maxMem=4042889625
16/09/07 19:11:01 INFO MemoryStore: Block broadcast_1065 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:01 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19025059, maxMem=4042889625
16/09/07 19:11:01 INFO MemoryStore: Block broadcast_1065_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:01 INFO BlockManagerInfo: Added broadcast_1065_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:01 INFO SparkContext: Created broadcast 1065 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:01 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 704 (MapPartitionsRDD[2143] at repartition at SparkUtils.java:190)
16/09/07 19:11:01 INFO TaskSchedulerImpl: Adding task set 704.0 with 8 tasks
16/09/07 19:11:01 INFO TaskSetManager: Starting task 0.0 in stage 704.0 (TID 5632, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 1.0 in stage 704.0 (TID 5633, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 2.0 in stage 704.0 (TID 5634, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 3.0 in stage 704.0 (TID 5635, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 4.0 in stage 704.0 (TID 5636, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 5.0 in stage 704.0 (TID 5637, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 6.0 in stage 704.0 (TID 5638, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 7.0 in stage 704.0 (TID 5639, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:01 INFO Executor: Running task 1.0 in stage 704.0 (TID 5633)
16/09/07 19:11:01 INFO Executor: Running task 4.0 in stage 704.0 (TID 5636)
16/09/07 19:11:01 INFO Executor: Running task 3.0 in stage 704.0 (TID 5635)
16/09/07 19:11:01 INFO Executor: Running task 7.0 in stage 704.0 (TID 5639)
16/09/07 19:11:01 INFO Executor: Running task 6.0 in stage 704.0 (TID 5638)
16/09/07 19:11:01 INFO Executor: Running task 0.0 in stage 704.0 (TID 5632)
16/09/07 19:11:01 INFO Executor: Running task 2.0 in stage 704.0 (TID 5634)
16/09/07 19:11:01 INFO Executor: Running task 5.0 in stage 704.0 (TID 5637)
16/09/07 19:11:01 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:01 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:01 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:01 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:01 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:01 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:01 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:01 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:01 INFO Executor: Finished task 6.0 in stage 704.0 (TID 5638). 2260 bytes result sent to driver
16/09/07 19:11:01 INFO Executor: Finished task 0.0 in stage 704.0 (TID 5632). 2260 bytes result sent to driver
16/09/07 19:11:01 INFO TaskSetManager: Finished task 0.0 in stage 704.0 (TID 5632) in 3 ms on localhost (1/8)
16/09/07 19:11:01 INFO TaskSetManager: Finished task 6.0 in stage 704.0 (TID 5638) in 2 ms on localhost (2/8)
16/09/07 19:11:01 INFO Executor: Finished task 5.0 in stage 704.0 (TID 5637). 2260 bytes result sent to driver
16/09/07 19:11:01 INFO Executor: Finished task 4.0 in stage 704.0 (TID 5636). 2260 bytes result sent to driver
16/09/07 19:11:01 INFO Executor: Finished task 2.0 in stage 704.0 (TID 5634). 2260 bytes result sent to driver
16/09/07 19:11:01 INFO Executor: Finished task 1.0 in stage 704.0 (TID 5633). 2260 bytes result sent to driver
16/09/07 19:11:01 INFO Executor: Finished task 3.0 in stage 704.0 (TID 5635). 2260 bytes result sent to driver
16/09/07 19:11:01 INFO Executor: Finished task 7.0 in stage 704.0 (TID 5639). 2260 bytes result sent to driver
16/09/07 19:11:01 INFO TaskSetManager: Finished task 5.0 in stage 704.0 (TID 5637) in 3 ms on localhost (3/8)
16/09/07 19:11:01 INFO TaskSetManager: Finished task 2.0 in stage 704.0 (TID 5634) in 3 ms on localhost (4/8)
16/09/07 19:11:01 INFO TaskSetManager: Finished task 4.0 in stage 704.0 (TID 5636) in 3 ms on localhost (5/8)
16/09/07 19:11:01 INFO TaskSetManager: Finished task 3.0 in stage 704.0 (TID 5635) in 3 ms on localhost (6/8)
16/09/07 19:11:01 INFO TaskSetManager: Finished task 1.0 in stage 704.0 (TID 5633) in 3 ms on localhost (7/8)
16/09/07 19:11:01 INFO TaskSetManager: Finished task 7.0 in stage 704.0 (TID 5639) in 3 ms on localhost (8/8)
16/09/07 19:11:01 INFO TaskSchedulerImpl: Removed TaskSet 704.0, whose tasks have all completed, from pool 
16/09/07 19:11:01 INFO DAGScheduler: ShuffleMapStage 704 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:01 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:01 INFO DAGScheduler: running: Set()
16/09/07 19:11:01 INFO DAGScheduler: waiting: Set(ResultStage 705)
16/09/07 19:11:01 INFO DAGScheduler: failed: Set()
16/09/07 19:11:01 INFO DAGScheduler: Missing parents for ResultStage 705: List()
16/09/07 19:11:01 INFO DAGScheduler: Submitting ResultStage 705 (MapPartitionsRDD[2147] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:01 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19027623, maxMem=4042889625
16/09/07 19:11:01 INFO MemoryStore: Block broadcast_1066 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:01 INFO MemoryStore: ensureFreeSpace(2742) called with curMem=19032719, maxMem=4042889625
16/09/07 19:11:01 INFO MemoryStore: Block broadcast_1066_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:01 INFO BlockManagerInfo: Added broadcast_1066_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:01 INFO SparkContext: Created broadcast 1066 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:01 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 705 (MapPartitionsRDD[2147] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:01 INFO TaskSchedulerImpl: Adding task set 705.0 with 8 tasks
16/09/07 19:11:01 INFO TaskSetManager: Starting task 0.0 in stage 705.0 (TID 5640, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 1.0 in stage 705.0 (TID 5641, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 2.0 in stage 705.0 (TID 5642, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 3.0 in stage 705.0 (TID 5643, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 4.0 in stage 705.0 (TID 5644, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 5.0 in stage 705.0 (TID 5645, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 6.0 in stage 705.0 (TID 5646, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:01 INFO TaskSetManager: Starting task 7.0 in stage 705.0 (TID 5647, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:01 INFO Executor: Running task 0.0 in stage 705.0 (TID 5640)
16/09/07 19:11:01 INFO Executor: Running task 5.0 in stage 705.0 (TID 5645)
16/09/07 19:11:01 INFO Executor: Running task 4.0 in stage 705.0 (TID 5644)
16/09/07 19:11:01 INFO Executor: Running task 1.0 in stage 705.0 (TID 5641)
16/09/07 19:11:01 INFO Executor: Running task 7.0 in stage 705.0 (TID 5647)
16/09/07 19:11:01 INFO Executor: Running task 3.0 in stage 705.0 (TID 5643)
16/09/07 19:11:01 INFO Executor: Running task 2.0 in stage 705.0 (TID 5642)
16/09/07 19:11:01 INFO Executor: Running task 6.0 in stage 705.0 (TID 5646)
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:01 INFO Executor: Finished task 5.0 in stage 705.0 (TID 5645). 2160 bytes result sent to driver
16/09/07 19:11:01 INFO Executor: Finished task 2.0 in stage 705.0 (TID 5642). 2160 bytes result sent to driver
16/09/07 19:11:01 INFO TaskSetManager: Finished task 5.0 in stage 705.0 (TID 5645) in 2 ms on localhost (1/8)
16/09/07 19:11:01 INFO TaskSetManager: Finished task 2.0 in stage 705.0 (TID 5642) in 2 ms on localhost (2/8)
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:01 INFO Executor: Finished task 3.0 in stage 705.0 (TID 5643). 2160 bytes result sent to driver
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:01 INFO TaskSetManager: Finished task 3.0 in stage 705.0 (TID 5643) in 3 ms on localhost (3/8)
16/09/07 19:11:01 INFO Executor: Finished task 1.0 in stage 705.0 (TID 5641). 2160 bytes result sent to driver
16/09/07 19:11:01 INFO TaskSetManager: Finished task 1.0 in stage 705.0 (TID 5641) in 3 ms on localhost (4/8)
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:01 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:01 INFO Executor: Finished task 4.0 in stage 705.0 (TID 5644). 2160 bytes result sent to driver
16/09/07 19:11:01 INFO TaskSetManager: Finished task 4.0 in stage 705.0 (TID 5644) in 4 ms on localhost (5/8)
16/09/07 19:11:01 INFO BlockManagerInfo: Removed broadcast_1065_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:02 INFO Executor: Finished task 6.0 in stage 705.0 (TID 5646). 3485274 bytes result sent to driver
16/09/07 19:11:02 INFO Executor: Finished task 7.0 in stage 705.0 (TID 5647). 3485274 bytes result sent to driver
16/09/07 19:11:02 INFO Executor: Finished task 0.0 in stage 705.0 (TID 5640). 3485274 bytes result sent to driver
16/09/07 19:11:02 INFO TaskSetManager: Finished task 6.0 in stage 705.0 (TID 5646) in 881 ms on localhost (6/8)
16/09/07 19:11:02 INFO TaskSetManager: Finished task 7.0 in stage 705.0 (TID 5647) in 896 ms on localhost (7/8)
16/09/07 19:11:02 INFO TaskSetManager: Finished task 0.0 in stage 705.0 (TID 5640) in 897 ms on localhost (8/8)
16/09/07 19:11:02 INFO TaskSchedulerImpl: Removed TaskSet 705.0, whose tasks have all completed, from pool 
16/09/07 19:11:02 INFO DAGScheduler: ResultStage 705 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.897 s
16/09/07 19:11:02 INFO DAGScheduler: Job 357 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.905125 s
16/09/07 19:11:02 INFO ParameterAveragingTrainingMaster: Completed training of split 36 of 78
16/09/07 19:11:02 INFO ParameterAveragingTrainingMaster: Starting training of split 37 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:02 INFO BlockManagerInfo: Removed broadcast_1066_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:02 INFO ContextCleaner: Cleaned accumulator 706
16/09/07 19:11:02 INFO ContextCleaner: Cleaned accumulator 705
16/09/07 19:11:02 INFO BlockManagerInfo: Removed broadcast_1064_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:02 INFO ContextCleaner: Cleaned shuffle 347
16/09/07 19:11:02 INFO MemoryStore: ensureFreeSpace(18497368) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:02 INFO MemoryStore: Block broadcast_1067 stored as values in memory (estimated size 17.6 MB, free 3.7 GB)
16/09/07 19:11:02 INFO MemoryStore: ensureFreeSpace(168077) called with curMem=19041192, maxMem=4042889625
16/09/07 19:11:02 INFO MemoryStore: Block broadcast_1067_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:02 INFO BlockManagerInfo: Added broadcast_1067_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:02 INFO SparkContext: Created broadcast 1067 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:02 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:02 INFO DAGScheduler: Registering RDD 2148 (repartition at SparkUtils.java:190)
16/09/07 19:11:02 INFO DAGScheduler: Got job 358 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:02 INFO DAGScheduler: Final stage: ResultStage 707(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:02 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 706)
16/09/07 19:11:02 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 706)
16/09/07 19:11:02 INFO DAGScheduler: Submitting ShuffleMapStage 706 (MapPartitionsRDD[2148] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:02 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19209269, maxMem=4042889625
16/09/07 19:11:02 INFO MemoryStore: Block broadcast_1068 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:02 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=19213733, maxMem=4042889625
16/09/07 19:11:02 INFO MemoryStore: Block broadcast_1068_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:02 INFO BlockManagerInfo: Added broadcast_1068_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:02 INFO SparkContext: Created broadcast 1068 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:02 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 706 (MapPartitionsRDD[2148] at repartition at SparkUtils.java:190)
16/09/07 19:11:02 INFO TaskSchedulerImpl: Adding task set 706.0 with 8 tasks
16/09/07 19:11:02 INFO TaskSetManager: Starting task 0.0 in stage 706.0 (TID 5648, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 1.0 in stage 706.0 (TID 5649, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 2.0 in stage 706.0 (TID 5650, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 3.0 in stage 706.0 (TID 5651, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 4.0 in stage 706.0 (TID 5652, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 5.0 in stage 706.0 (TID 5653, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 6.0 in stage 706.0 (TID 5654, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 7.0 in stage 706.0 (TID 5655, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:02 INFO Executor: Running task 1.0 in stage 706.0 (TID 5649)
16/09/07 19:11:02 INFO Executor: Running task 0.0 in stage 706.0 (TID 5648)
16/09/07 19:11:02 INFO Executor: Running task 3.0 in stage 706.0 (TID 5651)
16/09/07 19:11:02 INFO Executor: Running task 7.0 in stage 706.0 (TID 5655)
16/09/07 19:11:02 INFO Executor: Running task 6.0 in stage 706.0 (TID 5654)
16/09/07 19:11:02 INFO Executor: Running task 2.0 in stage 706.0 (TID 5650)
16/09/07 19:11:02 INFO Executor: Running task 5.0 in stage 706.0 (TID 5653)
16/09/07 19:11:02 INFO Executor: Running task 4.0 in stage 706.0 (TID 5652)
16/09/07 19:11:02 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:02 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:02 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:02 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:02 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:02 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:02 INFO Executor: Finished task 7.0 in stage 706.0 (TID 5655). 2260 bytes result sent to driver
16/09/07 19:11:02 INFO Executor: Finished task 1.0 in stage 706.0 (TID 5649). 2260 bytes result sent to driver
16/09/07 19:11:02 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:02 INFO Executor: Finished task 5.0 in stage 706.0 (TID 5653). 2260 bytes result sent to driver
16/09/07 19:11:02 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:02 INFO TaskSetManager: Finished task 7.0 in stage 706.0 (TID 5655) in 2 ms on localhost (1/8)
16/09/07 19:11:02 INFO TaskSetManager: Finished task 1.0 in stage 706.0 (TID 5649) in 3 ms on localhost (2/8)
16/09/07 19:11:02 INFO Executor: Finished task 3.0 in stage 706.0 (TID 5651). 2260 bytes result sent to driver
16/09/07 19:11:02 INFO TaskSetManager: Finished task 5.0 in stage 706.0 (TID 5653) in 2 ms on localhost (3/8)
16/09/07 19:11:02 INFO TaskSetManager: Finished task 3.0 in stage 706.0 (TID 5651) in 2 ms on localhost (4/8)
16/09/07 19:11:02 INFO Executor: Finished task 0.0 in stage 706.0 (TID 5648). 2260 bytes result sent to driver
16/09/07 19:11:02 INFO TaskSetManager: Finished task 0.0 in stage 706.0 (TID 5648) in 3 ms on localhost (5/8)
16/09/07 19:11:02 INFO Executor: Finished task 4.0 in stage 706.0 (TID 5652). 2260 bytes result sent to driver
16/09/07 19:11:02 INFO TaskSetManager: Finished task 4.0 in stage 706.0 (TID 5652) in 2 ms on localhost (6/8)
16/09/07 19:11:02 INFO Executor: Finished task 6.0 in stage 706.0 (TID 5654). 2260 bytes result sent to driver
16/09/07 19:11:02 INFO Executor: Finished task 2.0 in stage 706.0 (TID 5650). 2260 bytes result sent to driver
16/09/07 19:11:02 INFO TaskSetManager: Finished task 6.0 in stage 706.0 (TID 5654) in 3 ms on localhost (7/8)
16/09/07 19:11:02 INFO TaskSetManager: Finished task 2.0 in stage 706.0 (TID 5650) in 4 ms on localhost (8/8)
16/09/07 19:11:02 INFO TaskSchedulerImpl: Removed TaskSet 706.0, whose tasks have all completed, from pool 
16/09/07 19:11:02 INFO DAGScheduler: ShuffleMapStage 706 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:02 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:02 INFO DAGScheduler: running: Set()
16/09/07 19:11:02 INFO DAGScheduler: waiting: Set(ResultStage 707)
16/09/07 19:11:02 INFO DAGScheduler: failed: Set()
16/09/07 19:11:02 INFO DAGScheduler: Missing parents for ResultStage 707: List()
16/09/07 19:11:02 INFO DAGScheduler: Submitting ResultStage 707 (MapPartitionsRDD[2152] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:02 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19216295, maxMem=4042889625
16/09/07 19:11:02 INFO MemoryStore: Block broadcast_1069 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:02 INFO MemoryStore: ensureFreeSpace(2754) called with curMem=19221391, maxMem=4042889625
16/09/07 19:11:02 INFO MemoryStore: Block broadcast_1069_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:02 INFO BlockManagerInfo: Added broadcast_1069_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:02 INFO SparkContext: Created broadcast 1069 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:02 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 707 (MapPartitionsRDD[2152] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:02 INFO TaskSchedulerImpl: Adding task set 707.0 with 8 tasks
16/09/07 19:11:02 INFO TaskSetManager: Starting task 0.0 in stage 707.0 (TID 5656, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 1.0 in stage 707.0 (TID 5657, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 2.0 in stage 707.0 (TID 5658, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 3.0 in stage 707.0 (TID 5659, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 4.0 in stage 707.0 (TID 5660, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 5.0 in stage 707.0 (TID 5661, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 6.0 in stage 707.0 (TID 5662, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:02 INFO TaskSetManager: Starting task 7.0 in stage 707.0 (TID 5663, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:02 INFO Executor: Running task 1.0 in stage 707.0 (TID 5657)
16/09/07 19:11:02 INFO Executor: Running task 4.0 in stage 707.0 (TID 5660)
16/09/07 19:11:02 INFO Executor: Running task 5.0 in stage 707.0 (TID 5661)
16/09/07 19:11:02 INFO Executor: Running task 3.0 in stage 707.0 (TID 5659)
16/09/07 19:11:02 INFO Executor: Running task 0.0 in stage 707.0 (TID 5656)
16/09/07 19:11:02 INFO Executor: Running task 2.0 in stage 707.0 (TID 5658)
16/09/07 19:11:02 INFO Executor: Running task 6.0 in stage 707.0 (TID 5662)
16/09/07 19:11:02 INFO Executor: Running task 7.0 in stage 707.0 (TID 5663)
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:02 INFO Executor: Finished task 1.0 in stage 707.0 (TID 5657). 2160 bytes result sent to driver
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:02 INFO TaskSetManager: Finished task 1.0 in stage 707.0 (TID 5657) in 2 ms on localhost (1/8)
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:02 INFO Executor: Finished task 3.0 in stage 707.0 (TID 5659). 2160 bytes result sent to driver
16/09/07 19:11:02 INFO Executor: Finished task 5.0 in stage 707.0 (TID 5661). 2160 bytes result sent to driver
16/09/07 19:11:02 INFO TaskSetManager: Finished task 3.0 in stage 707.0 (TID 5659) in 3 ms on localhost (2/8)
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:02 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:02 INFO Executor: Finished task 4.0 in stage 707.0 (TID 5660). 2160 bytes result sent to driver
16/09/07 19:11:02 INFO TaskSetManager: Finished task 5.0 in stage 707.0 (TID 5661) in 6 ms on localhost (3/8)
16/09/07 19:11:02 INFO TaskSetManager: Finished task 4.0 in stage 707.0 (TID 5660) in 6 ms on localhost (4/8)
16/09/07 19:11:02 INFO Executor: Finished task 2.0 in stage 707.0 (TID 5658). 2160 bytes result sent to driver
16/09/07 19:11:02 INFO TaskSetManager: Finished task 2.0 in stage 707.0 (TID 5658) in 7 ms on localhost (5/8)
16/09/07 19:11:02 INFO BlockManagerInfo: Removed broadcast_1068_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:03 INFO Executor: Finished task 0.0 in stage 707.0 (TID 5656). 3485274 bytes result sent to driver
16/09/07 19:11:03 INFO Executor: Finished task 7.0 in stage 707.0 (TID 5663). 3485274 bytes result sent to driver
16/09/07 19:11:03 INFO Executor: Finished task 6.0 in stage 707.0 (TID 5662). 3485274 bytes result sent to driver
16/09/07 19:11:03 INFO TaskSetManager: Finished task 0.0 in stage 707.0 (TID 5656) in 914 ms on localhost (6/8)
16/09/07 19:11:03 INFO TaskSetManager: Finished task 7.0 in stage 707.0 (TID 5663) in 920 ms on localhost (7/8)
16/09/07 19:11:03 INFO TaskSetManager: Finished task 6.0 in stage 707.0 (TID 5662) in 933 ms on localhost (8/8)
16/09/07 19:11:03 INFO DAGScheduler: ResultStage 707 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.933 s
16/09/07 19:11:03 INFO TaskSchedulerImpl: Removed TaskSet 707.0, whose tasks have all completed, from pool 
16/09/07 19:11:03 INFO DAGScheduler: Job 358 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.940089 s
16/09/07 19:11:03 INFO ParameterAveragingTrainingMaster: Completed training of split 37 of 78
16/09/07 19:11:03 INFO ParameterAveragingTrainingMaster: Starting training of split 38 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:03 INFO BlockManagerInfo: Removed broadcast_1069_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:03 INFO ContextCleaner: Cleaned accumulator 708
16/09/07 19:11:03 INFO ContextCleaner: Cleaned accumulator 707
16/09/07 19:11:03 INFO BlockManagerInfo: Removed broadcast_1067_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:03 INFO ContextCleaner: Cleaned shuffle 348
16/09/07 19:11:03 INFO MemoryStore: ensureFreeSpace(18520056) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:03 INFO MemoryStore: Block broadcast_1070 stored as values in memory (estimated size 17.7 MB, free 3.7 GB)
16/09/07 19:11:03 INFO MemoryStore: ensureFreeSpace(168051) called with curMem=19063880, maxMem=4042889625
16/09/07 19:11:03 INFO MemoryStore: Block broadcast_1070_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:03 INFO BlockManagerInfo: Added broadcast_1070_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:03 INFO SparkContext: Created broadcast 1070 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:03 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:03 INFO DAGScheduler: Registering RDD 2153 (repartition at SparkUtils.java:190)
16/09/07 19:11:03 INFO DAGScheduler: Got job 359 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:03 INFO DAGScheduler: Final stage: ResultStage 709(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:03 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 708)
16/09/07 19:11:03 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 708)
16/09/07 19:11:03 INFO DAGScheduler: Submitting ShuffleMapStage 708 (MapPartitionsRDD[2153] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:03 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19231931, maxMem=4042889625
16/09/07 19:11:03 INFO MemoryStore: Block broadcast_1071 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:03 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19236395, maxMem=4042889625
16/09/07 19:11:03 INFO MemoryStore: Block broadcast_1071_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:03 INFO BlockManagerInfo: Added broadcast_1071_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:03 INFO SparkContext: Created broadcast 1071 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:03 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 708 (MapPartitionsRDD[2153] at repartition at SparkUtils.java:190)
16/09/07 19:11:03 INFO TaskSchedulerImpl: Adding task set 708.0 with 8 tasks
16/09/07 19:11:03 INFO TaskSetManager: Starting task 0.0 in stage 708.0 (TID 5664, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 1.0 in stage 708.0 (TID 5665, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 2.0 in stage 708.0 (TID 5666, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 3.0 in stage 708.0 (TID 5667, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 4.0 in stage 708.0 (TID 5668, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 5.0 in stage 708.0 (TID 5669, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 6.0 in stage 708.0 (TID 5670, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 7.0 in stage 708.0 (TID 5671, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:03 INFO Executor: Running task 2.0 in stage 708.0 (TID 5666)
16/09/07 19:11:03 INFO Executor: Running task 0.0 in stage 708.0 (TID 5664)
16/09/07 19:11:03 INFO Executor: Running task 4.0 in stage 708.0 (TID 5668)
16/09/07 19:11:03 INFO Executor: Running task 3.0 in stage 708.0 (TID 5667)
16/09/07 19:11:03 INFO Executor: Running task 5.0 in stage 708.0 (TID 5669)
16/09/07 19:11:03 INFO Executor: Running task 1.0 in stage 708.0 (TID 5665)
16/09/07 19:11:03 INFO Executor: Running task 6.0 in stage 708.0 (TID 5670)
16/09/07 19:11:03 INFO Executor: Running task 7.0 in stage 708.0 (TID 5671)
16/09/07 19:11:03 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:03 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:03 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:03 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:03 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:03 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:03 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:03 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:03 INFO Executor: Finished task 0.0 in stage 708.0 (TID 5664). 2260 bytes result sent to driver
16/09/07 19:11:03 INFO Executor: Finished task 3.0 in stage 708.0 (TID 5667). 2260 bytes result sent to driver
16/09/07 19:11:03 INFO TaskSetManager: Finished task 0.0 in stage 708.0 (TID 5664) in 3 ms on localhost (1/8)
16/09/07 19:11:03 INFO TaskSetManager: Finished task 3.0 in stage 708.0 (TID 5667) in 3 ms on localhost (2/8)
16/09/07 19:11:03 INFO Executor: Finished task 1.0 in stage 708.0 (TID 5665). 2260 bytes result sent to driver
16/09/07 19:11:03 INFO Executor: Finished task 5.0 in stage 708.0 (TID 5669). 2260 bytes result sent to driver
16/09/07 19:11:03 INFO TaskSetManager: Finished task 1.0 in stage 708.0 (TID 5665) in 3 ms on localhost (3/8)
16/09/07 19:11:03 INFO TaskSetManager: Finished task 5.0 in stage 708.0 (TID 5669) in 3 ms on localhost (4/8)
16/09/07 19:11:03 INFO Executor: Finished task 2.0 in stage 708.0 (TID 5666). 2260 bytes result sent to driver
16/09/07 19:11:03 INFO TaskSetManager: Finished task 2.0 in stage 708.0 (TID 5666) in 4 ms on localhost (5/8)
16/09/07 19:11:03 INFO Executor: Finished task 4.0 in stage 708.0 (TID 5668). 2260 bytes result sent to driver
16/09/07 19:11:03 INFO Executor: Finished task 7.0 in stage 708.0 (TID 5671). 2260 bytes result sent to driver
16/09/07 19:11:03 INFO Executor: Finished task 6.0 in stage 708.0 (TID 5670). 2260 bytes result sent to driver
16/09/07 19:11:03 INFO TaskSetManager: Finished task 6.0 in stage 708.0 (TID 5670) in 4 ms on localhost (6/8)
16/09/07 19:11:03 INFO TaskSetManager: Finished task 7.0 in stage 708.0 (TID 5671) in 4 ms on localhost (7/8)
16/09/07 19:11:03 INFO TaskSetManager: Finished task 4.0 in stage 708.0 (TID 5668) in 4 ms on localhost (8/8)
16/09/07 19:11:03 INFO DAGScheduler: ShuffleMapStage 708 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:03 INFO TaskSchedulerImpl: Removed TaskSet 708.0, whose tasks have all completed, from pool 
16/09/07 19:11:03 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:03 INFO DAGScheduler: running: Set()
16/09/07 19:11:03 INFO DAGScheduler: waiting: Set(ResultStage 709)
16/09/07 19:11:03 INFO DAGScheduler: failed: Set()
16/09/07 19:11:03 INFO DAGScheduler: Missing parents for ResultStage 709: List()
16/09/07 19:11:03 INFO DAGScheduler: Submitting ResultStage 709 (MapPartitionsRDD[2157] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:03 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19238959, maxMem=4042889625
16/09/07 19:11:03 INFO MemoryStore: Block broadcast_1072 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:03 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19244055, maxMem=4042889625
16/09/07 19:11:03 INFO MemoryStore: Block broadcast_1072_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:03 INFO BlockManagerInfo: Added broadcast_1072_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:03 INFO SparkContext: Created broadcast 1072 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:03 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 709 (MapPartitionsRDD[2157] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:03 INFO TaskSchedulerImpl: Adding task set 709.0 with 8 tasks
16/09/07 19:11:03 INFO TaskSetManager: Starting task 0.0 in stage 709.0 (TID 5672, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 1.0 in stage 709.0 (TID 5673, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 2.0 in stage 709.0 (TID 5674, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 3.0 in stage 709.0 (TID 5675, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 4.0 in stage 709.0 (TID 5676, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 5.0 in stage 709.0 (TID 5677, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 6.0 in stage 709.0 (TID 5678, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:03 INFO TaskSetManager: Starting task 7.0 in stage 709.0 (TID 5679, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:03 INFO Executor: Running task 6.0 in stage 709.0 (TID 5678)
16/09/07 19:11:03 INFO Executor: Running task 7.0 in stage 709.0 (TID 5679)
16/09/07 19:11:03 INFO Executor: Running task 5.0 in stage 709.0 (TID 5677)
16/09/07 19:11:03 INFO Executor: Running task 0.0 in stage 709.0 (TID 5672)
16/09/07 19:11:03 INFO Executor: Running task 1.0 in stage 709.0 (TID 5673)
16/09/07 19:11:03 INFO Executor: Running task 4.0 in stage 709.0 (TID 5676)
16/09/07 19:11:03 INFO Executor: Running task 3.0 in stage 709.0 (TID 5675)
16/09/07 19:11:03 INFO Executor: Running task 2.0 in stage 709.0 (TID 5674)
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:03 INFO Executor: Finished task 1.0 in stage 709.0 (TID 5673). 2160 bytes result sent to driver
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:03 INFO Executor: Finished task 3.0 in stage 709.0 (TID 5675). 2160 bytes result sent to driver
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:03 INFO TaskSetManager: Finished task 1.0 in stage 709.0 (TID 5673) in 2 ms on localhost (1/8)
16/09/07 19:11:03 INFO Executor: Finished task 4.0 in stage 709.0 (TID 5676). 2160 bytes result sent to driver
16/09/07 19:11:03 INFO TaskSetManager: Finished task 3.0 in stage 709.0 (TID 5675) in 2 ms on localhost (2/8)
16/09/07 19:11:03 INFO TaskSetManager: Finished task 4.0 in stage 709.0 (TID 5676) in 2 ms on localhost (3/8)
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:03 INFO Executor: Finished task 5.0 in stage 709.0 (TID 5677). 2160 bytes result sent to driver
16/09/07 19:11:03 INFO TaskSetManager: Finished task 5.0 in stage 709.0 (TID 5677) in 4 ms on localhost (4/8)
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:03 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:03 INFO Executor: Finished task 2.0 in stage 709.0 (TID 5674). 2160 bytes result sent to driver
16/09/07 19:11:03 INFO TaskSetManager: Finished task 2.0 in stage 709.0 (TID 5674) in 10 ms on localhost (5/8)
16/09/07 19:11:04 INFO BlockManagerInfo: Removed broadcast_1071_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:04 INFO Executor: Finished task 6.0 in stage 709.0 (TID 5678). 3485274 bytes result sent to driver
16/09/07 19:11:04 INFO Executor: Finished task 7.0 in stage 709.0 (TID 5679). 3485274 bytes result sent to driver
16/09/07 19:11:04 INFO Executor: Finished task 0.0 in stage 709.0 (TID 5672). 3485274 bytes result sent to driver
16/09/07 19:11:04 INFO TaskSetManager: Finished task 6.0 in stage 709.0 (TID 5678) in 760 ms on localhost (6/8)
16/09/07 19:11:04 INFO TaskSetManager: Finished task 7.0 in stage 709.0 (TID 5679) in 775 ms on localhost (7/8)
16/09/07 19:11:04 INFO TaskSetManager: Finished task 0.0 in stage 709.0 (TID 5672) in 775 ms on localhost (8/8)
16/09/07 19:11:04 INFO TaskSchedulerImpl: Removed TaskSet 709.0, whose tasks have all completed, from pool 
16/09/07 19:11:04 INFO DAGScheduler: ResultStage 709 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.775 s
16/09/07 19:11:04 INFO DAGScheduler: Job 359 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.783922 s
16/09/07 19:11:04 INFO ParameterAveragingTrainingMaster: Completed training of split 38 of 78
16/09/07 19:11:04 INFO ParameterAveragingTrainingMaster: Starting training of split 39 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:04 INFO BlockManagerInfo: Removed broadcast_1072_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:04 INFO ContextCleaner: Cleaned accumulator 710
16/09/07 19:11:04 INFO ContextCleaner: Cleaned accumulator 709
16/09/07 19:11:04 INFO BlockManagerInfo: Removed broadcast_1070_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:04 INFO ContextCleaner: Cleaned shuffle 349
16/09/07 19:11:04 INFO MemoryStore: ensureFreeSpace(18534176) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:04 INFO MemoryStore: Block broadcast_1073 stored as values in memory (estimated size 17.7 MB, free 3.7 GB)
16/09/07 19:11:05 INFO MemoryStore: ensureFreeSpace(168022) called with curMem=19078000, maxMem=4042889625
16/09/07 19:11:05 INFO MemoryStore: Block broadcast_1073_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:05 INFO BlockManagerInfo: Added broadcast_1073_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:05 INFO SparkContext: Created broadcast 1073 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:05 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:05 INFO DAGScheduler: Registering RDD 2158 (repartition at SparkUtils.java:190)
16/09/07 19:11:05 INFO DAGScheduler: Got job 360 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:05 INFO DAGScheduler: Final stage: ResultStage 711(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:05 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 710)
16/09/07 19:11:05 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 710)
16/09/07 19:11:05 INFO DAGScheduler: Submitting ShuffleMapStage 710 (MapPartitionsRDD[2158] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:05 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19246022, maxMem=4042889625
16/09/07 19:11:05 INFO MemoryStore: Block broadcast_1074 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:05 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=19250486, maxMem=4042889625
16/09/07 19:11:05 INFO MemoryStore: Block broadcast_1074_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:05 INFO BlockManagerInfo: Added broadcast_1074_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:05 INFO SparkContext: Created broadcast 1074 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:05 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 710 (MapPartitionsRDD[2158] at repartition at SparkUtils.java:190)
16/09/07 19:11:05 INFO TaskSchedulerImpl: Adding task set 710.0 with 8 tasks
16/09/07 19:11:05 INFO TaskSetManager: Starting task 0.0 in stage 710.0 (TID 5680, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 1.0 in stage 710.0 (TID 5681, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 2.0 in stage 710.0 (TID 5682, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 3.0 in stage 710.0 (TID 5683, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 4.0 in stage 710.0 (TID 5684, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 5.0 in stage 710.0 (TID 5685, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 6.0 in stage 710.0 (TID 5686, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 7.0 in stage 710.0 (TID 5687, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:05 INFO Executor: Running task 1.0 in stage 710.0 (TID 5681)
16/09/07 19:11:05 INFO Executor: Running task 2.0 in stage 710.0 (TID 5682)
16/09/07 19:11:05 INFO Executor: Running task 3.0 in stage 710.0 (TID 5683)
16/09/07 19:11:05 INFO Executor: Running task 4.0 in stage 710.0 (TID 5684)
16/09/07 19:11:05 INFO Executor: Running task 5.0 in stage 710.0 (TID 5685)
16/09/07 19:11:05 INFO Executor: Running task 0.0 in stage 710.0 (TID 5680)
16/09/07 19:11:05 INFO Executor: Running task 6.0 in stage 710.0 (TID 5686)
16/09/07 19:11:05 INFO Executor: Running task 7.0 in stage 710.0 (TID 5687)
16/09/07 19:11:05 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:05 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:05 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:05 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:05 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:05 INFO Executor: Finished task 2.0 in stage 710.0 (TID 5682). 2260 bytes result sent to driver
16/09/07 19:11:05 INFO Executor: Finished task 3.0 in stage 710.0 (TID 5683). 2260 bytes result sent to driver
16/09/07 19:11:05 INFO TaskSetManager: Finished task 2.0 in stage 710.0 (TID 5682) in 3 ms on localhost (1/8)
16/09/07 19:11:05 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:05 INFO TaskSetManager: Finished task 3.0 in stage 710.0 (TID 5683) in 3 ms on localhost (2/8)
16/09/07 19:11:05 INFO Executor: Finished task 6.0 in stage 710.0 (TID 5686). 2260 bytes result sent to driver
16/09/07 19:11:05 INFO TaskSetManager: Finished task 6.0 in stage 710.0 (TID 5686) in 3 ms on localhost (3/8)
16/09/07 19:11:05 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:05 INFO Executor: Finished task 5.0 in stage 710.0 (TID 5685). 2260 bytes result sent to driver
16/09/07 19:11:05 INFO Executor: Finished task 0.0 in stage 710.0 (TID 5680). 2260 bytes result sent to driver
16/09/07 19:11:05 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:05 INFO TaskSetManager: Finished task 5.0 in stage 710.0 (TID 5685) in 3 ms on localhost (4/8)
16/09/07 19:11:05 INFO TaskSetManager: Finished task 0.0 in stage 710.0 (TID 5680) in 3 ms on localhost (5/8)
16/09/07 19:11:05 INFO Executor: Finished task 4.0 in stage 710.0 (TID 5684). 2260 bytes result sent to driver
16/09/07 19:11:05 INFO TaskSetManager: Finished task 4.0 in stage 710.0 (TID 5684) in 4 ms on localhost (6/8)
16/09/07 19:11:05 INFO Executor: Finished task 1.0 in stage 710.0 (TID 5681). 2260 bytes result sent to driver
16/09/07 19:11:05 INFO Executor: Finished task 7.0 in stage 710.0 (TID 5687). 2260 bytes result sent to driver
16/09/07 19:11:05 INFO TaskSetManager: Finished task 1.0 in stage 710.0 (TID 5681) in 4 ms on localhost (7/8)
16/09/07 19:11:05 INFO TaskSetManager: Finished task 7.0 in stage 710.0 (TID 5687) in 4 ms on localhost (8/8)
16/09/07 19:11:05 INFO TaskSchedulerImpl: Removed TaskSet 710.0, whose tasks have all completed, from pool 
16/09/07 19:11:05 INFO DAGScheduler: ShuffleMapStage 710 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:05 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:05 INFO DAGScheduler: running: Set()
16/09/07 19:11:05 INFO DAGScheduler: waiting: Set(ResultStage 711)
16/09/07 19:11:05 INFO DAGScheduler: failed: Set()
16/09/07 19:11:05 INFO DAGScheduler: Missing parents for ResultStage 711: List()
16/09/07 19:11:05 INFO DAGScheduler: Submitting ResultStage 711 (MapPartitionsRDD[2162] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:05 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19253048, maxMem=4042889625
16/09/07 19:11:05 INFO MemoryStore: Block broadcast_1075 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:05 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=19258144, maxMem=4042889625
16/09/07 19:11:05 INFO MemoryStore: Block broadcast_1075_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:05 INFO BlockManagerInfo: Added broadcast_1075_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:05 INFO SparkContext: Created broadcast 1075 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:05 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 711 (MapPartitionsRDD[2162] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:05 INFO TaskSchedulerImpl: Adding task set 711.0 with 8 tasks
16/09/07 19:11:05 INFO TaskSetManager: Starting task 0.0 in stage 711.0 (TID 5688, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 1.0 in stage 711.0 (TID 5689, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 2.0 in stage 711.0 (TID 5690, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 3.0 in stage 711.0 (TID 5691, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 4.0 in stage 711.0 (TID 5692, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 5.0 in stage 711.0 (TID 5693, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 6.0 in stage 711.0 (TID 5694, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:05 INFO TaskSetManager: Starting task 7.0 in stage 711.0 (TID 5695, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:05 INFO Executor: Running task 3.0 in stage 711.0 (TID 5691)
16/09/07 19:11:05 INFO Executor: Running task 1.0 in stage 711.0 (TID 5689)
16/09/07 19:11:05 INFO Executor: Running task 7.0 in stage 711.0 (TID 5695)
16/09/07 19:11:05 INFO Executor: Running task 0.0 in stage 711.0 (TID 5688)
16/09/07 19:11:05 INFO Executor: Running task 4.0 in stage 711.0 (TID 5692)
16/09/07 19:11:05 INFO Executor: Running task 6.0 in stage 711.0 (TID 5694)
16/09/07 19:11:05 INFO Executor: Running task 5.0 in stage 711.0 (TID 5693)
16/09/07 19:11:05 INFO Executor: Running task 2.0 in stage 711.0 (TID 5690)
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:05 INFO Executor: Finished task 1.0 in stage 711.0 (TID 5689). 2160 bytes result sent to driver
16/09/07 19:11:05 INFO Executor: Finished task 2.0 in stage 711.0 (TID 5690). 2160 bytes result sent to driver
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:05 INFO TaskSetManager: Finished task 2.0 in stage 711.0 (TID 5690) in 2 ms on localhost (1/8)
16/09/07 19:11:05 INFO Executor: Finished task 5.0 in stage 711.0 (TID 5693). 2160 bytes result sent to driver
16/09/07 19:11:05 INFO TaskSetManager: Finished task 1.0 in stage 711.0 (TID 5689) in 2 ms on localhost (2/8)
16/09/07 19:11:05 INFO TaskSetManager: Finished task 5.0 in stage 711.0 (TID 5693) in 2 ms on localhost (3/8)
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:05 INFO Executor: Finished task 3.0 in stage 711.0 (TID 5691). 2160 bytes result sent to driver
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:05 INFO TaskSetManager: Finished task 3.0 in stage 711.0 (TID 5691) in 3 ms on localhost (4/8)
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:05 INFO Executor: Finished task 4.0 in stage 711.0 (TID 5692). 2160 bytes result sent to driver
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:05 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:05 INFO TaskSetManager: Finished task 4.0 in stage 711.0 (TID 5692) in 3 ms on localhost (5/8)
16/09/07 19:11:05 INFO BlockManagerInfo: Removed broadcast_1074_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:05 INFO Executor: Finished task 0.0 in stage 711.0 (TID 5688). 3485274 bytes result sent to driver
16/09/07 19:11:05 INFO Executor: Finished task 6.0 in stage 711.0 (TID 5694). 3485274 bytes result sent to driver
16/09/07 19:11:05 INFO Executor: Finished task 7.0 in stage 711.0 (TID 5695). 3485274 bytes result sent to driver
16/09/07 19:11:05 INFO TaskSetManager: Finished task 0.0 in stage 711.0 (TID 5688) in 852 ms on localhost (6/8)
16/09/07 19:11:05 INFO TaskSetManager: Finished task 6.0 in stage 711.0 (TID 5694) in 867 ms on localhost (7/8)
16/09/07 19:11:05 INFO TaskSetManager: Finished task 7.0 in stage 711.0 (TID 5695) in 868 ms on localhost (8/8)
16/09/07 19:11:05 INFO TaskSchedulerImpl: Removed TaskSet 711.0, whose tasks have all completed, from pool 
16/09/07 19:11:05 INFO DAGScheduler: ResultStage 711 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.868 s
16/09/07 19:11:05 INFO DAGScheduler: Job 360 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.876519 s
16/09/07 19:11:05 INFO ParameterAveragingTrainingMaster: Completed training of split 39 of 78
16/09/07 19:11:05 INFO ParameterAveragingTrainingMaster: Starting training of split 40 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:06 INFO BlockManagerInfo: Removed broadcast_1075_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:06 INFO ContextCleaner: Cleaned accumulator 712
16/09/07 19:11:06 INFO ContextCleaner: Cleaned accumulator 711
16/09/07 19:11:06 INFO BlockManagerInfo: Removed broadcast_1073_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:06 INFO ContextCleaner: Cleaned shuffle 350
16/09/07 19:11:06 INFO MemoryStore: ensureFreeSpace(18588032) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:06 INFO MemoryStore: Block broadcast_1076 stored as values in memory (estimated size 17.7 MB, free 3.7 GB)
16/09/07 19:11:06 INFO MemoryStore: ensureFreeSpace(167963) called with curMem=19131856, maxMem=4042889625
16/09/07 19:11:06 INFO MemoryStore: Block broadcast_1076_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:06 INFO BlockManagerInfo: Added broadcast_1076_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:06 INFO SparkContext: Created broadcast 1076 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:06 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:06 INFO DAGScheduler: Registering RDD 2163 (repartition at SparkUtils.java:190)
16/09/07 19:11:06 INFO DAGScheduler: Got job 361 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:06 INFO DAGScheduler: Final stage: ResultStage 713(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:06 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 712)
16/09/07 19:11:06 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 712)
16/09/07 19:11:06 INFO DAGScheduler: Submitting ShuffleMapStage 712 (MapPartitionsRDD[2163] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:06 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19299819, maxMem=4042889625
16/09/07 19:11:06 INFO MemoryStore: Block broadcast_1077 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:06 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19304283, maxMem=4042889625
16/09/07 19:11:06 INFO MemoryStore: Block broadcast_1077_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:06 INFO BlockManagerInfo: Added broadcast_1077_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:06 INFO SparkContext: Created broadcast 1077 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:06 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 712 (MapPartitionsRDD[2163] at repartition at SparkUtils.java:190)
16/09/07 19:11:06 INFO TaskSchedulerImpl: Adding task set 712.0 with 8 tasks
16/09/07 19:11:06 INFO TaskSetManager: Starting task 0.0 in stage 712.0 (TID 5696, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 1.0 in stage 712.0 (TID 5697, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 2.0 in stage 712.0 (TID 5698, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 3.0 in stage 712.0 (TID 5699, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 4.0 in stage 712.0 (TID 5700, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 5.0 in stage 712.0 (TID 5701, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 6.0 in stage 712.0 (TID 5702, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 7.0 in stage 712.0 (TID 5703, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:06 INFO Executor: Running task 0.0 in stage 712.0 (TID 5696)
16/09/07 19:11:06 INFO Executor: Running task 3.0 in stage 712.0 (TID 5699)
16/09/07 19:11:06 INFO Executor: Running task 2.0 in stage 712.0 (TID 5698)
16/09/07 19:11:06 INFO Executor: Running task 5.0 in stage 712.0 (TID 5701)
16/09/07 19:11:06 INFO Executor: Running task 4.0 in stage 712.0 (TID 5700)
16/09/07 19:11:06 INFO Executor: Running task 7.0 in stage 712.0 (TID 5703)
16/09/07 19:11:06 INFO Executor: Running task 1.0 in stage 712.0 (TID 5697)
16/09/07 19:11:06 INFO Executor: Running task 6.0 in stage 712.0 (TID 5702)
16/09/07 19:11:06 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:06 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:06 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:06 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:06 INFO Executor: Finished task 0.0 in stage 712.0 (TID 5696). 2260 bytes result sent to driver
16/09/07 19:11:06 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:06 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:06 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:06 INFO Executor: Finished task 5.0 in stage 712.0 (TID 5701). 2260 bytes result sent to driver
16/09/07 19:11:06 INFO TaskSetManager: Finished task 0.0 in stage 712.0 (TID 5696) in 3 ms on localhost (1/8)
16/09/07 19:11:06 INFO Executor: Finished task 1.0 in stage 712.0 (TID 5697). 2260 bytes result sent to driver
16/09/07 19:11:06 INFO TaskSetManager: Finished task 5.0 in stage 712.0 (TID 5701) in 2 ms on localhost (2/8)
16/09/07 19:11:06 INFO TaskSetManager: Finished task 1.0 in stage 712.0 (TID 5697) in 3 ms on localhost (3/8)
16/09/07 19:11:06 INFO Executor: Finished task 7.0 in stage 712.0 (TID 5703). 2260 bytes result sent to driver
16/09/07 19:11:06 INFO TaskSetManager: Finished task 7.0 in stage 712.0 (TID 5703) in 2 ms on localhost (4/8)
16/09/07 19:11:06 INFO Executor: Finished task 3.0 in stage 712.0 (TID 5699). 2260 bytes result sent to driver
16/09/07 19:11:06 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:06 INFO TaskSetManager: Finished task 3.0 in stage 712.0 (TID 5699) in 3 ms on localhost (5/8)
16/09/07 19:11:06 INFO Executor: Finished task 2.0 in stage 712.0 (TID 5698). 2260 bytes result sent to driver
16/09/07 19:11:06 INFO Executor: Finished task 4.0 in stage 712.0 (TID 5700). 2260 bytes result sent to driver
16/09/07 19:11:06 INFO TaskSetManager: Finished task 2.0 in stage 712.0 (TID 5698) in 4 ms on localhost (6/8)
16/09/07 19:11:06 INFO TaskSetManager: Finished task 4.0 in stage 712.0 (TID 5700) in 3 ms on localhost (7/8)
16/09/07 19:11:06 INFO Executor: Finished task 6.0 in stage 712.0 (TID 5702). 2260 bytes result sent to driver
16/09/07 19:11:06 INFO TaskSetManager: Finished task 6.0 in stage 712.0 (TID 5702) in 3 ms on localhost (8/8)
16/09/07 19:11:06 INFO TaskSchedulerImpl: Removed TaskSet 712.0, whose tasks have all completed, from pool 
16/09/07 19:11:06 INFO DAGScheduler: ShuffleMapStage 712 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:11:06 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:06 INFO DAGScheduler: running: Set()
16/09/07 19:11:06 INFO DAGScheduler: waiting: Set(ResultStage 713)
16/09/07 19:11:06 INFO DAGScheduler: failed: Set()
16/09/07 19:11:06 INFO DAGScheduler: Missing parents for ResultStage 713: List()
16/09/07 19:11:06 INFO DAGScheduler: Submitting ResultStage 713 (MapPartitionsRDD[2167] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:06 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19306847, maxMem=4042889625
16/09/07 19:11:06 INFO MemoryStore: Block broadcast_1078 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:06 INFO MemoryStore: ensureFreeSpace(2742) called with curMem=19311943, maxMem=4042889625
16/09/07 19:11:06 INFO MemoryStore: Block broadcast_1078_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:06 INFO BlockManagerInfo: Added broadcast_1078_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:06 INFO SparkContext: Created broadcast 1078 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:06 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 713 (MapPartitionsRDD[2167] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:06 INFO TaskSchedulerImpl: Adding task set 713.0 with 8 tasks
16/09/07 19:11:06 INFO TaskSetManager: Starting task 0.0 in stage 713.0 (TID 5704, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 1.0 in stage 713.0 (TID 5705, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 2.0 in stage 713.0 (TID 5706, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 3.0 in stage 713.0 (TID 5707, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 4.0 in stage 713.0 (TID 5708, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 5.0 in stage 713.0 (TID 5709, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 6.0 in stage 713.0 (TID 5710, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:06 INFO TaskSetManager: Starting task 7.0 in stage 713.0 (TID 5711, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:06 INFO Executor: Running task 1.0 in stage 713.0 (TID 5705)
16/09/07 19:11:06 INFO Executor: Running task 2.0 in stage 713.0 (TID 5706)
16/09/07 19:11:06 INFO Executor: Running task 3.0 in stage 713.0 (TID 5707)
16/09/07 19:11:06 INFO Executor: Running task 4.0 in stage 713.0 (TID 5708)
16/09/07 19:11:06 INFO Executor: Running task 5.0 in stage 713.0 (TID 5709)
16/09/07 19:11:06 INFO Executor: Running task 7.0 in stage 713.0 (TID 5711)
16/09/07 19:11:06 INFO Executor: Running task 0.0 in stage 713.0 (TID 5704)
16/09/07 19:11:06 INFO Executor: Running task 6.0 in stage 713.0 (TID 5710)
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:06 INFO Executor: Finished task 5.0 in stage 713.0 (TID 5709). 2160 bytes result sent to driver
16/09/07 19:11:06 INFO Executor: Finished task 2.0 in stage 713.0 (TID 5706). 2160 bytes result sent to driver
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:06 INFO TaskSetManager: Finished task 5.0 in stage 713.0 (TID 5709) in 2 ms on localhost (1/8)
16/09/07 19:11:06 INFO TaskSetManager: Finished task 2.0 in stage 713.0 (TID 5706) in 2 ms on localhost (2/8)
16/09/07 19:11:06 INFO Executor: Finished task 4.0 in stage 713.0 (TID 5708). 2160 bytes result sent to driver
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:06 INFO TaskSetManager: Finished task 4.0 in stage 713.0 (TID 5708) in 2 ms on localhost (3/8)
16/09/07 19:11:06 INFO Executor: Finished task 1.0 in stage 713.0 (TID 5705). 2160 bytes result sent to driver
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:06 INFO TaskSetManager: Finished task 1.0 in stage 713.0 (TID 5705) in 3 ms on localhost (4/8)
16/09/07 19:11:06 INFO Executor: Finished task 3.0 in stage 713.0 (TID 5707). 2160 bytes result sent to driver
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:06 INFO TaskSetManager: Finished task 3.0 in stage 713.0 (TID 5707) in 3 ms on localhost (5/8)
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:06 INFO BlockManagerInfo: Removed broadcast_1077_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:06 INFO Executor: Finished task 6.0 in stage 713.0 (TID 5710). 3485274 bytes result sent to driver
16/09/07 19:11:06 INFO TaskSetManager: Finished task 6.0 in stage 713.0 (TID 5710) in 782 ms on localhost (6/8)
16/09/07 19:11:07 INFO Executor: Finished task 7.0 in stage 713.0 (TID 5711). 3485274 bytes result sent to driver
16/09/07 19:11:07 INFO Executor: Finished task 0.0 in stage 713.0 (TID 5704). 3485274 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 0.0 in stage 713.0 (TID 5704) in 837 ms on localhost (7/8)
16/09/07 19:11:07 INFO TaskSetManager: Finished task 7.0 in stage 713.0 (TID 5711) in 838 ms on localhost (8/8)
16/09/07 19:11:07 INFO TaskSchedulerImpl: Removed TaskSet 713.0, whose tasks have all completed, from pool 
16/09/07 19:11:07 INFO DAGScheduler: ResultStage 713 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.838 s
16/09/07 19:11:07 INFO DAGScheduler: Job 361 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.904149 s
16/09/07 19:11:07 INFO ParameterAveragingTrainingMaster: Completed training of split 40 of 78
16/09/07 19:11:07 INFO ParameterAveragingTrainingMaster: Starting training of split 41 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:07 INFO MemoryStore: ensureFreeSpace(18644296) called with curMem=19307657, maxMem=4042889625
16/09/07 19:11:07 INFO MemoryStore: Block broadcast_1079 stored as values in memory (estimated size 17.8 MB, free 3.7 GB)
16/09/07 19:11:07 INFO BlockManagerInfo: Removed broadcast_1078_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:07 INFO ContextCleaner: Cleaned accumulator 714
16/09/07 19:11:07 INFO ContextCleaner: Cleaned accumulator 713
16/09/07 19:11:07 INFO BlockManagerInfo: Removed broadcast_1076_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:07 INFO ContextCleaner: Cleaned shuffle 351
16/09/07 19:11:07 INFO MemoryStore: ensureFreeSpace(167941) called with curMem=19188120, maxMem=4042889625
16/09/07 19:11:07 INFO MemoryStore: Block broadcast_1079_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:07 INFO BlockManagerInfo: Added broadcast_1079_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:07 INFO SparkContext: Created broadcast 1079 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:07 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:07 INFO DAGScheduler: Registering RDD 2168 (repartition at SparkUtils.java:190)
16/09/07 19:11:07 INFO DAGScheduler: Got job 362 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:07 INFO DAGScheduler: Final stage: ResultStage 715(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:07 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 714)
16/09/07 19:11:07 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 714)
16/09/07 19:11:07 INFO DAGScheduler: Submitting ShuffleMapStage 714 (MapPartitionsRDD[2168] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:07 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19356061, maxMem=4042889625
16/09/07 19:11:07 INFO MemoryStore: Block broadcast_1080 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:07 INFO MemoryStore: ensureFreeSpace(2557) called with curMem=19360525, maxMem=4042889625
16/09/07 19:11:07 INFO MemoryStore: Block broadcast_1080_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:07 INFO BlockManagerInfo: Added broadcast_1080_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:07 INFO SparkContext: Created broadcast 1080 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:07 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 714 (MapPartitionsRDD[2168] at repartition at SparkUtils.java:190)
16/09/07 19:11:07 INFO TaskSchedulerImpl: Adding task set 714.0 with 8 tasks
16/09/07 19:11:07 INFO TaskSetManager: Starting task 0.0 in stage 714.0 (TID 5712, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 1.0 in stage 714.0 (TID 5713, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 2.0 in stage 714.0 (TID 5714, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 3.0 in stage 714.0 (TID 5715, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 4.0 in stage 714.0 (TID 5716, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 5.0 in stage 714.0 (TID 5717, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 6.0 in stage 714.0 (TID 5718, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 7.0 in stage 714.0 (TID 5719, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:07 INFO Executor: Running task 1.0 in stage 714.0 (TID 5713)
16/09/07 19:11:07 INFO Executor: Running task 0.0 in stage 714.0 (TID 5712)
16/09/07 19:11:07 INFO Executor: Running task 5.0 in stage 714.0 (TID 5717)
16/09/07 19:11:07 INFO Executor: Running task 4.0 in stage 714.0 (TID 5716)
16/09/07 19:11:07 INFO Executor: Running task 3.0 in stage 714.0 (TID 5715)
16/09/07 19:11:07 INFO Executor: Running task 6.0 in stage 714.0 (TID 5718)
16/09/07 19:11:07 INFO Executor: Running task 2.0 in stage 714.0 (TID 5714)
16/09/07 19:11:07 INFO Executor: Running task 7.0 in stage 714.0 (TID 5719)
16/09/07 19:11:07 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:07 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:07 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:07 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:07 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:07 INFO Executor: Finished task 1.0 in stage 714.0 (TID 5713). 2260 bytes result sent to driver
16/09/07 19:11:07 INFO Executor: Finished task 0.0 in stage 714.0 (TID 5712). 2260 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 1.0 in stage 714.0 (TID 5713) in 3 ms on localhost (1/8)
16/09/07 19:11:07 INFO TaskSetManager: Finished task 0.0 in stage 714.0 (TID 5712) in 3 ms on localhost (2/8)
16/09/07 19:11:07 INFO Executor: Finished task 5.0 in stage 714.0 (TID 5717). 2260 bytes result sent to driver
16/09/07 19:11:07 INFO Executor: Finished task 7.0 in stage 714.0 (TID 5719). 2260 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 5.0 in stage 714.0 (TID 5717) in 2 ms on localhost (3/8)
16/09/07 19:11:07 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:07 INFO TaskSetManager: Finished task 7.0 in stage 714.0 (TID 5719) in 2 ms on localhost (4/8)
16/09/07 19:11:07 INFO Executor: Finished task 3.0 in stage 714.0 (TID 5715). 2260 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 3.0 in stage 714.0 (TID 5715) in 2 ms on localhost (5/8)
16/09/07 19:11:07 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:07 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:07 INFO Executor: Finished task 6.0 in stage 714.0 (TID 5718). 2260 bytes result sent to driver
16/09/07 19:11:07 INFO Executor: Finished task 4.0 in stage 714.0 (TID 5716). 2260 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 6.0 in stage 714.0 (TID 5718) in 3 ms on localhost (6/8)
16/09/07 19:11:07 INFO TaskSetManager: Finished task 4.0 in stage 714.0 (TID 5716) in 3 ms on localhost (7/8)
16/09/07 19:11:07 INFO Executor: Finished task 2.0 in stage 714.0 (TID 5714). 2260 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 2.0 in stage 714.0 (TID 5714) in 5 ms on localhost (8/8)
16/09/07 19:11:07 INFO TaskSchedulerImpl: Removed TaskSet 714.0, whose tasks have all completed, from pool 
16/09/07 19:11:07 INFO DAGScheduler: ShuffleMapStage 714 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:11:07 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:07 INFO DAGScheduler: running: Set()
16/09/07 19:11:07 INFO DAGScheduler: waiting: Set(ResultStage 715)
16/09/07 19:11:07 INFO DAGScheduler: failed: Set()
16/09/07 19:11:07 INFO DAGScheduler: Missing parents for ResultStage 715: List()
16/09/07 19:11:07 INFO DAGScheduler: Submitting ResultStage 715 (MapPartitionsRDD[2172] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:07 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19363082, maxMem=4042889625
16/09/07 19:11:07 INFO MemoryStore: Block broadcast_1081 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:07 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19368178, maxMem=4042889625
16/09/07 19:11:07 INFO MemoryStore: Block broadcast_1081_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:07 INFO BlockManagerInfo: Added broadcast_1081_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:07 INFO SparkContext: Created broadcast 1081 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:07 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 715 (MapPartitionsRDD[2172] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:07 INFO TaskSchedulerImpl: Adding task set 715.0 with 8 tasks
16/09/07 19:11:07 INFO TaskSetManager: Starting task 0.0 in stage 715.0 (TID 5720, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 1.0 in stage 715.0 (TID 5721, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 2.0 in stage 715.0 (TID 5722, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 3.0 in stage 715.0 (TID 5723, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 4.0 in stage 715.0 (TID 5724, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 5.0 in stage 715.0 (TID 5725, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 6.0 in stage 715.0 (TID 5726, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:07 INFO TaskSetManager: Starting task 7.0 in stage 715.0 (TID 5727, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:07 INFO Executor: Running task 0.0 in stage 715.0 (TID 5720)
16/09/07 19:11:07 INFO Executor: Running task 5.0 in stage 715.0 (TID 5725)
16/09/07 19:11:07 INFO Executor: Running task 4.0 in stage 715.0 (TID 5724)
16/09/07 19:11:07 INFO Executor: Running task 1.0 in stage 715.0 (TID 5721)
16/09/07 19:11:07 INFO Executor: Running task 6.0 in stage 715.0 (TID 5726)
16/09/07 19:11:07 INFO Executor: Running task 2.0 in stage 715.0 (TID 5722)
16/09/07 19:11:07 INFO Executor: Running task 3.0 in stage 715.0 (TID 5723)
16/09/07 19:11:07 INFO Executor: Running task 7.0 in stage 715.0 (TID 5727)
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:07 INFO Executor: Finished task 3.0 in stage 715.0 (TID 5723). 2160 bytes result sent to driver
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:07 INFO Executor: Finished task 2.0 in stage 715.0 (TID 5722). 2160 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 3.0 in stage 715.0 (TID 5723) in 2 ms on localhost (1/8)
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:07 INFO TaskSetManager: Finished task 2.0 in stage 715.0 (TID 5722) in 2 ms on localhost (2/8)
16/09/07 19:11:07 INFO Executor: Finished task 1.0 in stage 715.0 (TID 5721). 2160 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 1.0 in stage 715.0 (TID 5721) in 3 ms on localhost (3/8)
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:07 INFO Executor: Finished task 4.0 in stage 715.0 (TID 5724). 2160 bytes result sent to driver
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:07 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:07 INFO TaskSetManager: Finished task 4.0 in stage 715.0 (TID 5724) in 3 ms on localhost (4/8)
16/09/07 19:11:07 INFO Executor: Finished task 5.0 in stage 715.0 (TID 5725). 2160 bytes result sent to driver
16/09/07 19:11:07 INFO TaskSetManager: Finished task 5.0 in stage 715.0 (TID 5725) in 4 ms on localhost (5/8)
16/09/07 19:11:07 INFO BlockManagerInfo: Removed broadcast_1080_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:08 INFO Executor: Finished task 7.0 in stage 715.0 (TID 5727). 3485274 bytes result sent to driver
16/09/07 19:11:08 INFO TaskSetManager: Finished task 7.0 in stage 715.0 (TID 5727) in 780 ms on localhost (6/8)
16/09/07 19:11:08 INFO Executor: Finished task 6.0 in stage 715.0 (TID 5726). 3485274 bytes result sent to driver
16/09/07 19:11:08 INFO Executor: Finished task 0.0 in stage 715.0 (TID 5720). 3485274 bytes result sent to driver
16/09/07 19:11:08 INFO TaskSetManager: Finished task 0.0 in stage 715.0 (TID 5720) in 886 ms on localhost (7/8)
16/09/07 19:11:08 INFO TaskSetManager: Finished task 6.0 in stage 715.0 (TID 5726) in 886 ms on localhost (8/8)
16/09/07 19:11:08 INFO TaskSchedulerImpl: Removed TaskSet 715.0, whose tasks have all completed, from pool 
16/09/07 19:11:08 INFO DAGScheduler: ResultStage 715 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.886 s
16/09/07 19:11:08 INFO DAGScheduler: Job 362 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.894083 s
16/09/07 19:11:08 INFO ParameterAveragingTrainingMaster: Completed training of split 41 of 78
16/09/07 19:11:08 INFO ParameterAveragingTrainingMaster: Starting training of split 42 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:08 INFO MemoryStore: ensureFreeSpace(18649416) called with curMem=19363900, maxMem=4042889625
16/09/07 19:11:08 INFO MemoryStore: Block broadcast_1082 stored as values in memory (estimated size 17.8 MB, free 3.7 GB)
16/09/07 19:11:08 INFO MemoryStore: ensureFreeSpace(167953) called with curMem=38013316, maxMem=4042889625
16/09/07 19:11:08 INFO MemoryStore: Block broadcast_1082_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:08 INFO BlockManagerInfo: Added broadcast_1082_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:08 INFO SparkContext: Created broadcast 1082 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:08 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:08 INFO DAGScheduler: Registering RDD 2173 (repartition at SparkUtils.java:190)
16/09/07 19:11:08 INFO DAGScheduler: Got job 363 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:08 INFO DAGScheduler: Final stage: ResultStage 717(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:08 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 716)
16/09/07 19:11:08 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 716)
16/09/07 19:11:08 INFO DAGScheduler: Submitting ShuffleMapStage 716 (MapPartitionsRDD[2173] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:08 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=38181269, maxMem=4042889625
16/09/07 19:11:08 INFO MemoryStore: Block broadcast_1083 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:08 INFO MemoryStore: ensureFreeSpace(2561) called with curMem=38185733, maxMem=4042889625
16/09/07 19:11:08 INFO MemoryStore: Block broadcast_1083_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:08 INFO BlockManagerInfo: Added broadcast_1083_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:08 INFO SparkContext: Created broadcast 1083 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:08 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 716 (MapPartitionsRDD[2173] at repartition at SparkUtils.java:190)
16/09/07 19:11:08 INFO TaskSchedulerImpl: Adding task set 716.0 with 8 tasks
16/09/07 19:11:08 INFO TaskSetManager: Starting task 0.0 in stage 716.0 (TID 5728, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 1.0 in stage 716.0 (TID 5729, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 2.0 in stage 716.0 (TID 5730, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 3.0 in stage 716.0 (TID 5731, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 4.0 in stage 716.0 (TID 5732, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 5.0 in stage 716.0 (TID 5733, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 6.0 in stage 716.0 (TID 5734, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 7.0 in stage 716.0 (TID 5735, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:08 INFO Executor: Running task 2.0 in stage 716.0 (TID 5730)
16/09/07 19:11:08 INFO Executor: Running task 3.0 in stage 716.0 (TID 5731)
16/09/07 19:11:08 INFO Executor: Running task 7.0 in stage 716.0 (TID 5735)
16/09/07 19:11:08 INFO Executor: Running task 4.0 in stage 716.0 (TID 5732)
16/09/07 19:11:08 INFO Executor: Running task 0.0 in stage 716.0 (TID 5728)
16/09/07 19:11:08 INFO Executor: Running task 5.0 in stage 716.0 (TID 5733)
16/09/07 19:11:08 INFO Executor: Running task 1.0 in stage 716.0 (TID 5729)
16/09/07 19:11:08 INFO Executor: Running task 6.0 in stage 716.0 (TID 5734)
16/09/07 19:11:08 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:08 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:08 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:08 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:08 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:08 INFO Executor: Finished task 2.0 in stage 716.0 (TID 5730). 2260 bytes result sent to driver
16/09/07 19:11:08 INFO Executor: Finished task 4.0 in stage 716.0 (TID 5732). 2260 bytes result sent to driver
16/09/07 19:11:08 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:08 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:08 INFO TaskSetManager: Finished task 2.0 in stage 716.0 (TID 5730) in 2 ms on localhost (1/8)
16/09/07 19:11:08 INFO TaskSetManager: Finished task 4.0 in stage 716.0 (TID 5732) in 2 ms on localhost (2/8)
16/09/07 19:11:08 INFO Executor: Finished task 6.0 in stage 716.0 (TID 5734). 2260 bytes result sent to driver
16/09/07 19:11:08 INFO Executor: Finished task 1.0 in stage 716.0 (TID 5729). 2260 bytes result sent to driver
16/09/07 19:11:08 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:08 INFO TaskSetManager: Finished task 1.0 in stage 716.0 (TID 5729) in 3 ms on localhost (3/8)
16/09/07 19:11:08 INFO TaskSetManager: Finished task 6.0 in stage 716.0 (TID 5734) in 3 ms on localhost (4/8)
16/09/07 19:11:08 INFO Executor: Finished task 0.0 in stage 716.0 (TID 5728). 2260 bytes result sent to driver
16/09/07 19:11:08 INFO Executor: Finished task 5.0 in stage 716.0 (TID 5733). 2260 bytes result sent to driver
16/09/07 19:11:08 INFO Executor: Finished task 3.0 in stage 716.0 (TID 5731). 2260 bytes result sent to driver
16/09/07 19:11:08 INFO TaskSetManager: Finished task 0.0 in stage 716.0 (TID 5728) in 3 ms on localhost (5/8)
16/09/07 19:11:08 INFO TaskSetManager: Finished task 5.0 in stage 716.0 (TID 5733) in 3 ms on localhost (6/8)
16/09/07 19:11:08 INFO TaskSetManager: Finished task 3.0 in stage 716.0 (TID 5731) in 3 ms on localhost (7/8)
16/09/07 19:11:08 INFO Executor: Finished task 7.0 in stage 716.0 (TID 5735). 2260 bytes result sent to driver
16/09/07 19:11:08 INFO TaskSetManager: Finished task 7.0 in stage 716.0 (TID 5735) in 4 ms on localhost (8/8)
16/09/07 19:11:08 INFO DAGScheduler: ShuffleMapStage 716 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:08 INFO TaskSchedulerImpl: Removed TaskSet 716.0, whose tasks have all completed, from pool 
16/09/07 19:11:08 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:08 INFO DAGScheduler: running: Set()
16/09/07 19:11:08 INFO DAGScheduler: waiting: Set(ResultStage 717)
16/09/07 19:11:08 INFO DAGScheduler: failed: Set()
16/09/07 19:11:08 INFO DAGScheduler: Missing parents for ResultStage 717: List()
16/09/07 19:11:08 INFO DAGScheduler: Submitting ResultStage 717 (MapPartitionsRDD[2177] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:08 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=38188294, maxMem=4042889625
16/09/07 19:11:08 INFO MemoryStore: Block broadcast_1084 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:08 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=38193390, maxMem=4042889625
16/09/07 19:11:08 INFO MemoryStore: Block broadcast_1084_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:08 INFO BlockManagerInfo: Added broadcast_1084_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:08 INFO SparkContext: Created broadcast 1084 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:08 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 717 (MapPartitionsRDD[2177] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:08 INFO TaskSchedulerImpl: Adding task set 717.0 with 8 tasks
16/09/07 19:11:08 INFO TaskSetManager: Starting task 0.0 in stage 717.0 (TID 5736, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 1.0 in stage 717.0 (TID 5737, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 2.0 in stage 717.0 (TID 5738, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 3.0 in stage 717.0 (TID 5739, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 4.0 in stage 717.0 (TID 5740, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 5.0 in stage 717.0 (TID 5741, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 6.0 in stage 717.0 (TID 5742, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:08 INFO TaskSetManager: Starting task 7.0 in stage 717.0 (TID 5743, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:08 INFO Executor: Running task 0.0 in stage 717.0 (TID 5736)
16/09/07 19:11:08 INFO Executor: Running task 6.0 in stage 717.0 (TID 5742)
16/09/07 19:11:08 INFO Executor: Running task 7.0 in stage 717.0 (TID 5743)
16/09/07 19:11:08 INFO Executor: Running task 5.0 in stage 717.0 (TID 5741)
16/09/07 19:11:08 INFO Executor: Running task 1.0 in stage 717.0 (TID 5737)
16/09/07 19:11:08 INFO Executor: Running task 2.0 in stage 717.0 (TID 5738)
16/09/07 19:11:08 INFO Executor: Running task 4.0 in stage 717.0 (TID 5740)
16/09/07 19:11:08 INFO Executor: Running task 3.0 in stage 717.0 (TID 5739)
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:08 INFO Executor: Finished task 4.0 in stage 717.0 (TID 5740). 2160 bytes result sent to driver
16/09/07 19:11:08 INFO Executor: Finished task 3.0 in stage 717.0 (TID 5739). 2160 bytes result sent to driver
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:08 INFO TaskSetManager: Finished task 4.0 in stage 717.0 (TID 5740) in 2 ms on localhost (1/8)
16/09/07 19:11:08 INFO Executor: Finished task 1.0 in stage 717.0 (TID 5737). 2160 bytes result sent to driver
16/09/07 19:11:08 INFO Executor: Finished task 2.0 in stage 717.0 (TID 5738). 2160 bytes result sent to driver
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:08 INFO TaskSetManager: Finished task 3.0 in stage 717.0 (TID 5739) in 3 ms on localhost (2/8)
16/09/07 19:11:08 INFO TaskSetManager: Finished task 1.0 in stage 717.0 (TID 5737) in 3 ms on localhost (3/8)
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:08 INFO TaskSetManager: Finished task 2.0 in stage 717.0 (TID 5738) in 3 ms on localhost (4/8)
16/09/07 19:11:08 INFO Executor: Finished task 5.0 in stage 717.0 (TID 5741). 2160 bytes result sent to driver
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:08 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:08 INFO TaskSetManager: Finished task 5.0 in stage 717.0 (TID 5741) in 2 ms on localhost (5/8)
16/09/07 19:11:08 INFO BlockManagerInfo: Removed broadcast_1083_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:08 INFO BlockManagerInfo: Removed broadcast_1081_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:08 INFO ContextCleaner: Cleaned accumulator 716
16/09/07 19:11:08 INFO ContextCleaner: Cleaned accumulator 715
16/09/07 19:11:08 INFO BlockManagerInfo: Removed broadcast_1079_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:08 INFO ContextCleaner: Cleaned shuffle 352
16/09/07 19:11:09 INFO Executor: Finished task 6.0 in stage 717.0 (TID 5742). 3485274 bytes result sent to driver
16/09/07 19:11:09 INFO Executor: Finished task 7.0 in stage 717.0 (TID 5743). 3485274 bytes result sent to driver
16/09/07 19:11:09 INFO Executor: Finished task 0.0 in stage 717.0 (TID 5736). 3485274 bytes result sent to driver
16/09/07 19:11:09 INFO TaskSetManager: Finished task 0.0 in stage 717.0 (TID 5736) in 790 ms on localhost (6/8)
16/09/07 19:11:09 INFO TaskSetManager: Finished task 6.0 in stage 717.0 (TID 5742) in 801 ms on localhost (7/8)
16/09/07 19:11:09 INFO TaskSetManager: Finished task 7.0 in stage 717.0 (TID 5743) in 802 ms on localhost (8/8)
16/09/07 19:11:09 INFO TaskSchedulerImpl: Removed TaskSet 717.0, whose tasks have all completed, from pool 
16/09/07 19:11:09 INFO DAGScheduler: ResultStage 717 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.803 s
16/09/07 19:11:09 INFO DAGScheduler: Job 363 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.810618 s
16/09/07 19:11:09 INFO ParameterAveragingTrainingMaster: Completed training of split 42 of 78
16/09/07 19:11:09 INFO ParameterAveragingTrainingMaster: Starting training of split 43 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:09 INFO BlockManagerInfo: Removed broadcast_1084_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:09 INFO ContextCleaner: Cleaned accumulator 718
16/09/07 19:11:09 INFO ContextCleaner: Cleaned accumulator 717
16/09/07 19:11:09 INFO BlockManagerInfo: Removed broadcast_1082_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:09 INFO ContextCleaner: Cleaned shuffle 353
16/09/07 19:11:09 INFO MemoryStore: ensureFreeSpace(18660112) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:09 INFO MemoryStore: Block broadcast_1085 stored as values in memory (estimated size 17.8 MB, free 3.7 GB)
16/09/07 19:11:09 INFO MemoryStore: ensureFreeSpace(167936) called with curMem=19203936, maxMem=4042889625
16/09/07 19:11:09 INFO MemoryStore: Block broadcast_1085_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:09 INFO BlockManagerInfo: Added broadcast_1085_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:09 INFO SparkContext: Created broadcast 1085 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:09 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:09 INFO DAGScheduler: Registering RDD 2178 (repartition at SparkUtils.java:190)
16/09/07 19:11:09 INFO DAGScheduler: Got job 364 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:09 INFO DAGScheduler: Final stage: ResultStage 719(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:09 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 718)
16/09/07 19:11:09 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 718)
16/09/07 19:11:09 INFO DAGScheduler: Submitting ShuffleMapStage 718 (MapPartitionsRDD[2178] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:09 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19371872, maxMem=4042889625
16/09/07 19:11:09 INFO MemoryStore: Block broadcast_1086 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:09 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19376336, maxMem=4042889625
16/09/07 19:11:09 INFO MemoryStore: Block broadcast_1086_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:09 INFO BlockManagerInfo: Added broadcast_1086_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:09 INFO SparkContext: Created broadcast 1086 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:09 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 718 (MapPartitionsRDD[2178] at repartition at SparkUtils.java:190)
16/09/07 19:11:09 INFO TaskSchedulerImpl: Adding task set 718.0 with 8 tasks
16/09/07 19:11:09 INFO TaskSetManager: Starting task 0.0 in stage 718.0 (TID 5744, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 1.0 in stage 718.0 (TID 5745, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 2.0 in stage 718.0 (TID 5746, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 3.0 in stage 718.0 (TID 5747, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 4.0 in stage 718.0 (TID 5748, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 5.0 in stage 718.0 (TID 5749, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 6.0 in stage 718.0 (TID 5750, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 7.0 in stage 718.0 (TID 5751, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:09 INFO Executor: Running task 0.0 in stage 718.0 (TID 5744)
16/09/07 19:11:09 INFO Executor: Running task 1.0 in stage 718.0 (TID 5745)
16/09/07 19:11:09 INFO Executor: Running task 5.0 in stage 718.0 (TID 5749)
16/09/07 19:11:09 INFO Executor: Running task 7.0 in stage 718.0 (TID 5751)
16/09/07 19:11:09 INFO Executor: Running task 6.0 in stage 718.0 (TID 5750)
16/09/07 19:11:09 INFO Executor: Running task 4.0 in stage 718.0 (TID 5748)
16/09/07 19:11:09 INFO Executor: Running task 2.0 in stage 718.0 (TID 5746)
16/09/07 19:11:09 INFO Executor: Running task 3.0 in stage 718.0 (TID 5747)
16/09/07 19:11:09 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:09 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:09 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:09 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:09 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:09 INFO Executor: Finished task 2.0 in stage 718.0 (TID 5746). 2260 bytes result sent to driver
16/09/07 19:11:09 INFO Executor: Finished task 0.0 in stage 718.0 (TID 5744). 2260 bytes result sent to driver
16/09/07 19:11:09 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:09 INFO Executor: Finished task 6.0 in stage 718.0 (TID 5750). 2260 bytes result sent to driver
16/09/07 19:11:09 INFO TaskSetManager: Finished task 0.0 in stage 718.0 (TID 5744) in 3 ms on localhost (1/8)
16/09/07 19:11:09 INFO TaskSetManager: Finished task 2.0 in stage 718.0 (TID 5746) in 3 ms on localhost (2/8)
16/09/07 19:11:09 INFO Executor: Finished task 3.0 in stage 718.0 (TID 5747). 2260 bytes result sent to driver
16/09/07 19:11:09 INFO TaskSetManager: Finished task 6.0 in stage 718.0 (TID 5750) in 2 ms on localhost (3/8)
16/09/07 19:11:09 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:09 INFO Executor: Finished task 1.0 in stage 718.0 (TID 5745). 2260 bytes result sent to driver
16/09/07 19:11:09 INFO Executor: Finished task 4.0 in stage 718.0 (TID 5748). 2260 bytes result sent to driver
16/09/07 19:11:09 INFO TaskSetManager: Finished task 3.0 in stage 718.0 (TID 5747) in 2 ms on localhost (4/8)
16/09/07 19:11:09 INFO TaskSetManager: Finished task 1.0 in stage 718.0 (TID 5745) in 3 ms on localhost (5/8)
16/09/07 19:11:09 INFO TaskSetManager: Finished task 4.0 in stage 718.0 (TID 5748) in 2 ms on localhost (6/8)
16/09/07 19:11:09 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:09 INFO Executor: Finished task 7.0 in stage 718.0 (TID 5751). 2260 bytes result sent to driver
16/09/07 19:11:09 INFO TaskSetManager: Finished task 7.0 in stage 718.0 (TID 5751) in 3 ms on localhost (7/8)
16/09/07 19:11:09 INFO Executor: Finished task 5.0 in stage 718.0 (TID 5749). 2260 bytes result sent to driver
16/09/07 19:11:09 INFO TaskSetManager: Finished task 5.0 in stage 718.0 (TID 5749) in 3 ms on localhost (8/8)
16/09/07 19:11:09 INFO TaskSchedulerImpl: Removed TaskSet 718.0, whose tasks have all completed, from pool 
16/09/07 19:11:09 INFO DAGScheduler: ShuffleMapStage 718 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:09 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:09 INFO DAGScheduler: running: Set()
16/09/07 19:11:09 INFO DAGScheduler: waiting: Set(ResultStage 719)
16/09/07 19:11:09 INFO DAGScheduler: failed: Set()
16/09/07 19:11:09 INFO DAGScheduler: Missing parents for ResultStage 719: List()
16/09/07 19:11:09 INFO DAGScheduler: Submitting ResultStage 719 (MapPartitionsRDD[2182] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:09 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19378900, maxMem=4042889625
16/09/07 19:11:09 INFO MemoryStore: Block broadcast_1087 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:09 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19383996, maxMem=4042889625
16/09/07 19:11:09 INFO MemoryStore: Block broadcast_1087_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:09 INFO BlockManagerInfo: Added broadcast_1087_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:09 INFO SparkContext: Created broadcast 1087 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:09 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 719 (MapPartitionsRDD[2182] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:09 INFO TaskSchedulerImpl: Adding task set 719.0 with 8 tasks
16/09/07 19:11:09 INFO TaskSetManager: Starting task 0.0 in stage 719.0 (TID 5752, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 1.0 in stage 719.0 (TID 5753, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 2.0 in stage 719.0 (TID 5754, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 3.0 in stage 719.0 (TID 5755, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 4.0 in stage 719.0 (TID 5756, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 5.0 in stage 719.0 (TID 5757, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 6.0 in stage 719.0 (TID 5758, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:09 INFO TaskSetManager: Starting task 7.0 in stage 719.0 (TID 5759, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:09 INFO Executor: Running task 0.0 in stage 719.0 (TID 5752)
16/09/07 19:11:09 INFO Executor: Running task 7.0 in stage 719.0 (TID 5759)
16/09/07 19:11:09 INFO Executor: Running task 3.0 in stage 719.0 (TID 5755)
16/09/07 19:11:09 INFO Executor: Running task 4.0 in stage 719.0 (TID 5756)
16/09/07 19:11:09 INFO Executor: Running task 5.0 in stage 719.0 (TID 5757)
16/09/07 19:11:09 INFO Executor: Running task 2.0 in stage 719.0 (TID 5754)
16/09/07 19:11:09 INFO Executor: Running task 1.0 in stage 719.0 (TID 5753)
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:09 INFO Executor: Finished task 5.0 in stage 719.0 (TID 5757). 2160 bytes result sent to driver
16/09/07 19:11:09 INFO Executor: Running task 6.0 in stage 719.0 (TID 5758)
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:09 INFO Executor: Finished task 1.0 in stage 719.0 (TID 5753). 2160 bytes result sent to driver
16/09/07 19:11:09 INFO TaskSetManager: Finished task 5.0 in stage 719.0 (TID 5757) in 2 ms on localhost (1/8)
16/09/07 19:11:09 INFO Executor: Finished task 2.0 in stage 719.0 (TID 5754). 2160 bytes result sent to driver
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:09 INFO Executor: Finished task 3.0 in stage 719.0 (TID 5755). 2160 bytes result sent to driver
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:09 INFO Executor: Finished task 4.0 in stage 719.0 (TID 5756). 2160 bytes result sent to driver
16/09/07 19:11:09 INFO TaskSetManager: Finished task 1.0 in stage 719.0 (TID 5753) in 3 ms on localhost (2/8)
16/09/07 19:11:09 INFO TaskSetManager: Finished task 2.0 in stage 719.0 (TID 5754) in 3 ms on localhost (3/8)
16/09/07 19:11:09 INFO TaskSetManager: Finished task 3.0 in stage 719.0 (TID 5755) in 3 ms on localhost (4/8)
16/09/07 19:11:09 INFO TaskSetManager: Finished task 4.0 in stage 719.0 (TID 5756) in 4 ms on localhost (5/8)
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:09 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:09 INFO BlockManagerInfo: Removed broadcast_1086_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:10 INFO Executor: Finished task 0.0 in stage 719.0 (TID 5752). 3485274 bytes result sent to driver
16/09/07 19:11:10 INFO Executor: Finished task 7.0 in stage 719.0 (TID 5759). 3485274 bytes result sent to driver
16/09/07 19:11:10 INFO Executor: Finished task 6.0 in stage 719.0 (TID 5758). 3485274 bytes result sent to driver
16/09/07 19:11:10 INFO TaskSetManager: Finished task 7.0 in stage 719.0 (TID 5759) in 912 ms on localhost (6/8)
16/09/07 19:11:10 INFO TaskSetManager: Finished task 0.0 in stage 719.0 (TID 5752) in 924 ms on localhost (7/8)
16/09/07 19:11:10 INFO TaskSetManager: Finished task 6.0 in stage 719.0 (TID 5758) in 926 ms on localhost (8/8)
16/09/07 19:11:10 INFO TaskSchedulerImpl: Removed TaskSet 719.0, whose tasks have all completed, from pool 
16/09/07 19:11:10 INFO DAGScheduler: ResultStage 719 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.926 s
16/09/07 19:11:10 INFO DAGScheduler: Job 364 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.934104 s
16/09/07 19:11:10 INFO ParameterAveragingTrainingMaster: Completed training of split 43 of 78
16/09/07 19:11:10 INFO ParameterAveragingTrainingMaster: Starting training of split 44 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:10 INFO MemoryStore: ensureFreeSpace(18674632) called with curMem=19379711, maxMem=4042889625
16/09/07 19:11:10 INFO MemoryStore: Block broadcast_1088 stored as values in memory (estimated size 17.8 MB, free 3.7 GB)
16/09/07 19:11:10 INFO MemoryStore: ensureFreeSpace(167991) called with curMem=38054343, maxMem=4042889625
16/09/07 19:11:10 INFO MemoryStore: Block broadcast_1088_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:10 INFO BlockManagerInfo: Added broadcast_1088_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:10 INFO SparkContext: Created broadcast 1088 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:10 INFO BlockManagerInfo: Removed broadcast_1087_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:10 INFO ContextCleaner: Cleaned accumulator 720
16/09/07 19:11:10 INFO ContextCleaner: Cleaned accumulator 719
16/09/07 19:11:10 INFO BlockManagerInfo: Removed broadcast_1085_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:10 INFO ContextCleaner: Cleaned shuffle 354
16/09/07 19:11:10 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:10 INFO DAGScheduler: Registering RDD 2183 (repartition at SparkUtils.java:190)
16/09/07 19:11:10 INFO DAGScheduler: Got job 365 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:10 INFO DAGScheduler: Final stage: ResultStage 721(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:10 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 720)
16/09/07 19:11:10 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 720)
16/09/07 19:11:10 INFO DAGScheduler: Submitting ShuffleMapStage 720 (MapPartitionsRDD[2183] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:10 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19386447, maxMem=4042889625
16/09/07 19:11:10 INFO MemoryStore: Block broadcast_1089 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:10 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19390911, maxMem=4042889625
16/09/07 19:11:10 INFO MemoryStore: Block broadcast_1089_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:10 INFO BlockManagerInfo: Added broadcast_1089_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:10 INFO SparkContext: Created broadcast 1089 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:10 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 720 (MapPartitionsRDD[2183] at repartition at SparkUtils.java:190)
16/09/07 19:11:10 INFO TaskSchedulerImpl: Adding task set 720.0 with 8 tasks
16/09/07 19:11:10 INFO TaskSetManager: Starting task 0.0 in stage 720.0 (TID 5760, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 1.0 in stage 720.0 (TID 5761, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 2.0 in stage 720.0 (TID 5762, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 3.0 in stage 720.0 (TID 5763, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 4.0 in stage 720.0 (TID 5764, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 5.0 in stage 720.0 (TID 5765, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 6.0 in stage 720.0 (TID 5766, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 7.0 in stage 720.0 (TID 5767, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:10 INFO Executor: Running task 0.0 in stage 720.0 (TID 5760)
16/09/07 19:11:10 INFO Executor: Running task 5.0 in stage 720.0 (TID 5765)
16/09/07 19:11:10 INFO Executor: Running task 1.0 in stage 720.0 (TID 5761)
16/09/07 19:11:10 INFO Executor: Running task 6.0 in stage 720.0 (TID 5766)
16/09/07 19:11:10 INFO Executor: Running task 7.0 in stage 720.0 (TID 5767)
16/09/07 19:11:10 INFO Executor: Running task 2.0 in stage 720.0 (TID 5762)
16/09/07 19:11:10 INFO Executor: Running task 3.0 in stage 720.0 (TID 5763)
16/09/07 19:11:10 INFO Executor: Running task 4.0 in stage 720.0 (TID 5764)
16/09/07 19:11:10 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:10 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:10 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:10 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:10 INFO Executor: Finished task 7.0 in stage 720.0 (TID 5767). 2260 bytes result sent to driver
16/09/07 19:11:10 INFO Executor: Finished task 5.0 in stage 720.0 (TID 5765). 2260 bytes result sent to driver
16/09/07 19:11:10 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:10 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:10 INFO Executor: Finished task 4.0 in stage 720.0 (TID 5764). 2260 bytes result sent to driver
16/09/07 19:11:10 INFO TaskSetManager: Finished task 7.0 in stage 720.0 (TID 5767) in 2 ms on localhost (1/8)
16/09/07 19:11:10 INFO TaskSetManager: Finished task 5.0 in stage 720.0 (TID 5765) in 2 ms on localhost (2/8)
16/09/07 19:11:10 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:10 INFO Executor: Finished task 3.0 in stage 720.0 (TID 5763). 2260 bytes result sent to driver
16/09/07 19:11:10 INFO Executor: Finished task 1.0 in stage 720.0 (TID 5761). 2260 bytes result sent to driver
16/09/07 19:11:10 INFO TaskSetManager: Finished task 3.0 in stage 720.0 (TID 5763) in 3 ms on localhost (3/8)
16/09/07 19:11:10 INFO TaskSetManager: Finished task 4.0 in stage 720.0 (TID 5764) in 3 ms on localhost (4/8)
16/09/07 19:11:10 INFO TaskSetManager: Finished task 1.0 in stage 720.0 (TID 5761) in 3 ms on localhost (5/8)
16/09/07 19:11:10 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:10 INFO Executor: Finished task 2.0 in stage 720.0 (TID 5762). 2260 bytes result sent to driver
16/09/07 19:11:10 INFO TaskSetManager: Finished task 2.0 in stage 720.0 (TID 5762) in 3 ms on localhost (6/8)
16/09/07 19:11:10 INFO Executor: Finished task 0.0 in stage 720.0 (TID 5760). 2260 bytes result sent to driver
16/09/07 19:11:10 INFO Executor: Finished task 6.0 in stage 720.0 (TID 5766). 2260 bytes result sent to driver
16/09/07 19:11:10 INFO TaskSetManager: Finished task 0.0 in stage 720.0 (TID 5760) in 4 ms on localhost (7/8)
16/09/07 19:11:10 INFO TaskSetManager: Finished task 6.0 in stage 720.0 (TID 5766) in 4 ms on localhost (8/8)
16/09/07 19:11:10 INFO TaskSchedulerImpl: Removed TaskSet 720.0, whose tasks have all completed, from pool 
16/09/07 19:11:10 INFO DAGScheduler: ShuffleMapStage 720 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:10 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:10 INFO DAGScheduler: running: Set()
16/09/07 19:11:10 INFO DAGScheduler: waiting: Set(ResultStage 721)
16/09/07 19:11:10 INFO DAGScheduler: failed: Set()
16/09/07 19:11:10 INFO DAGScheduler: Missing parents for ResultStage 721: List()
16/09/07 19:11:10 INFO DAGScheduler: Submitting ResultStage 721 (MapPartitionsRDD[2187] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:10 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19393475, maxMem=4042889625
16/09/07 19:11:10 INFO MemoryStore: Block broadcast_1090 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:10 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19398571, maxMem=4042889625
16/09/07 19:11:10 INFO MemoryStore: Block broadcast_1090_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:10 INFO BlockManagerInfo: Added broadcast_1090_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:10 INFO SparkContext: Created broadcast 1090 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:10 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 721 (MapPartitionsRDD[2187] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:10 INFO TaskSchedulerImpl: Adding task set 721.0 with 8 tasks
16/09/07 19:11:10 INFO TaskSetManager: Starting task 0.0 in stage 721.0 (TID 5768, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 1.0 in stage 721.0 (TID 5769, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 2.0 in stage 721.0 (TID 5770, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 3.0 in stage 721.0 (TID 5771, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 4.0 in stage 721.0 (TID 5772, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 5.0 in stage 721.0 (TID 5773, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 6.0 in stage 721.0 (TID 5774, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:10 INFO TaskSetManager: Starting task 7.0 in stage 721.0 (TID 5775, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:10 INFO Executor: Running task 0.0 in stage 721.0 (TID 5768)
16/09/07 19:11:10 INFO Executor: Running task 6.0 in stage 721.0 (TID 5774)
16/09/07 19:11:10 INFO Executor: Running task 3.0 in stage 721.0 (TID 5771)
16/09/07 19:11:10 INFO Executor: Running task 2.0 in stage 721.0 (TID 5770)
16/09/07 19:11:10 INFO Executor: Running task 1.0 in stage 721.0 (TID 5769)
16/09/07 19:11:10 INFO Executor: Running task 5.0 in stage 721.0 (TID 5773)
16/09/07 19:11:10 INFO Executor: Running task 7.0 in stage 721.0 (TID 5775)
16/09/07 19:11:10 INFO Executor: Running task 4.0 in stage 721.0 (TID 5772)
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:10 INFO Executor: Finished task 4.0 in stage 721.0 (TID 5772). 2160 bytes result sent to driver
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:10 INFO TaskSetManager: Finished task 4.0 in stage 721.0 (TID 5772) in 2 ms on localhost (1/8)
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:10 INFO Executor: Finished task 5.0 in stage 721.0 (TID 5773). 2160 bytes result sent to driver
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:10 INFO Executor: Finished task 3.0 in stage 721.0 (TID 5771). 2160 bytes result sent to driver
16/09/07 19:11:10 INFO TaskSetManager: Finished task 5.0 in stage 721.0 (TID 5773) in 2 ms on localhost (2/8)
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:10 INFO Executor: Finished task 1.0 in stage 721.0 (TID 5769). 2160 bytes result sent to driver
16/09/07 19:11:10 INFO TaskSetManager: Finished task 3.0 in stage 721.0 (TID 5771) in 3 ms on localhost (3/8)
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:10 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:10 INFO TaskSetManager: Finished task 1.0 in stage 721.0 (TID 5769) in 3 ms on localhost (4/8)
16/09/07 19:11:10 INFO Executor: Finished task 2.0 in stage 721.0 (TID 5770). 2160 bytes result sent to driver
16/09/07 19:11:10 INFO TaskSetManager: Finished task 2.0 in stage 721.0 (TID 5770) in 3 ms on localhost (5/8)
16/09/07 19:11:11 INFO BlockManagerInfo: Removed broadcast_1089_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:11 INFO Executor: Finished task 6.0 in stage 721.0 (TID 5774). 3485274 bytes result sent to driver
16/09/07 19:11:11 INFO TaskSetManager: Finished task 6.0 in stage 721.0 (TID 5774) in 767 ms on localhost (6/8)
16/09/07 19:11:11 INFO Executor: Finished task 0.0 in stage 721.0 (TID 5768). 3485274 bytes result sent to driver
16/09/07 19:11:11 INFO Executor: Finished task 7.0 in stage 721.0 (TID 5775). 3485274 bytes result sent to driver
16/09/07 19:11:11 INFO TaskSetManager: Finished task 7.0 in stage 721.0 (TID 5775) in 812 ms on localhost (7/8)
16/09/07 19:11:11 INFO TaskSetManager: Finished task 0.0 in stage 721.0 (TID 5768) in 812 ms on localhost (8/8)
16/09/07 19:11:11 INFO TaskSchedulerImpl: Removed TaskSet 721.0, whose tasks have all completed, from pool 
16/09/07 19:11:11 INFO DAGScheduler: ResultStage 721 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.812 s
16/09/07 19:11:11 INFO DAGScheduler: Job 365 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.820500 s
16/09/07 19:11:11 INFO ParameterAveragingTrainingMaster: Completed training of split 44 of 78
16/09/07 19:11:11 INFO ParameterAveragingTrainingMaster: Starting training of split 45 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:11 INFO MemoryStore: ensureFreeSpace(18686544) called with curMem=19394286, maxMem=4042889625
16/09/07 19:11:11 INFO MemoryStore: Block broadcast_1091 stored as values in memory (estimated size 17.8 MB, free 3.7 GB)
16/09/07 19:11:11 INFO BlockManagerInfo: Removed broadcast_1090_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:11 INFO ContextCleaner: Cleaned accumulator 722
16/09/07 19:11:11 INFO ContextCleaner: Cleaned accumulator 721
16/09/07 19:11:11 INFO BlockManagerInfo: Removed broadcast_1088_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:11 INFO ContextCleaner: Cleaned shuffle 355
16/09/07 19:11:12 INFO MemoryStore: ensureFreeSpace(168022) called with curMem=19230368, maxMem=4042889625
16/09/07 19:11:12 INFO MemoryStore: Block broadcast_1091_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:12 INFO BlockManagerInfo: Added broadcast_1091_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:12 INFO SparkContext: Created broadcast 1091 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:12 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:12 INFO DAGScheduler: Registering RDD 2188 (repartition at SparkUtils.java:190)
16/09/07 19:11:12 INFO DAGScheduler: Got job 366 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:12 INFO DAGScheduler: Final stage: ResultStage 723(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:12 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 722)
16/09/07 19:11:12 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 722)
16/09/07 19:11:12 INFO DAGScheduler: Submitting ShuffleMapStage 722 (MapPartitionsRDD[2188] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:12 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19398390, maxMem=4042889625
16/09/07 19:11:12 INFO MemoryStore: Block broadcast_1092 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:12 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=19402854, maxMem=4042889625
16/09/07 19:11:12 INFO MemoryStore: Block broadcast_1092_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:12 INFO BlockManagerInfo: Added broadcast_1092_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:12 INFO SparkContext: Created broadcast 1092 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:12 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 722 (MapPartitionsRDD[2188] at repartition at SparkUtils.java:190)
16/09/07 19:11:12 INFO TaskSchedulerImpl: Adding task set 722.0 with 8 tasks
16/09/07 19:11:12 INFO TaskSetManager: Starting task 0.0 in stage 722.0 (TID 5776, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 1.0 in stage 722.0 (TID 5777, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 2.0 in stage 722.0 (TID 5778, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 3.0 in stage 722.0 (TID 5779, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 4.0 in stage 722.0 (TID 5780, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 5.0 in stage 722.0 (TID 5781, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 6.0 in stage 722.0 (TID 5782, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 7.0 in stage 722.0 (TID 5783, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:12 INFO Executor: Running task 1.0 in stage 722.0 (TID 5777)
16/09/07 19:11:12 INFO Executor: Running task 2.0 in stage 722.0 (TID 5778)
16/09/07 19:11:12 INFO Executor: Running task 0.0 in stage 722.0 (TID 5776)
16/09/07 19:11:12 INFO Executor: Running task 4.0 in stage 722.0 (TID 5780)
16/09/07 19:11:12 INFO Executor: Running task 5.0 in stage 722.0 (TID 5781)
16/09/07 19:11:12 INFO Executor: Running task 7.0 in stage 722.0 (TID 5783)
16/09/07 19:11:12 INFO Executor: Running task 6.0 in stage 722.0 (TID 5782)
16/09/07 19:11:12 INFO Executor: Running task 3.0 in stage 722.0 (TID 5779)
16/09/07 19:11:12 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:12 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:12 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:12 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:12 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:12 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:12 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:12 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:12 INFO Executor: Finished task 7.0 in stage 722.0 (TID 5783). 2260 bytes result sent to driver
16/09/07 19:11:12 INFO Executor: Finished task 4.0 in stage 722.0 (TID 5780). 2260 bytes result sent to driver
16/09/07 19:11:12 INFO Executor: Finished task 1.0 in stage 722.0 (TID 5777). 2260 bytes result sent to driver
16/09/07 19:11:12 INFO TaskSetManager: Finished task 4.0 in stage 722.0 (TID 5780) in 4 ms on localhost (1/8)
16/09/07 19:11:12 INFO Executor: Finished task 3.0 in stage 722.0 (TID 5779). 2260 bytes result sent to driver
16/09/07 19:11:12 INFO TaskSetManager: Finished task 7.0 in stage 722.0 (TID 5783) in 4 ms on localhost (2/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 1.0 in stage 722.0 (TID 5777) in 4 ms on localhost (3/8)
16/09/07 19:11:12 INFO Executor: Finished task 2.0 in stage 722.0 (TID 5778). 2260 bytes result sent to driver
16/09/07 19:11:12 INFO TaskSetManager: Finished task 3.0 in stage 722.0 (TID 5779) in 4 ms on localhost (4/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 2.0 in stage 722.0 (TID 5778) in 4 ms on localhost (5/8)
16/09/07 19:11:12 INFO Executor: Finished task 5.0 in stage 722.0 (TID 5781). 2260 bytes result sent to driver
16/09/07 19:11:12 INFO Executor: Finished task 0.0 in stage 722.0 (TID 5776). 2260 bytes result sent to driver
16/09/07 19:11:12 INFO Executor: Finished task 6.0 in stage 722.0 (TID 5782). 2260 bytes result sent to driver
16/09/07 19:11:12 INFO TaskSetManager: Finished task 0.0 in stage 722.0 (TID 5776) in 4 ms on localhost (6/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 5.0 in stage 722.0 (TID 5781) in 4 ms on localhost (7/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 6.0 in stage 722.0 (TID 5782) in 4 ms on localhost (8/8)
16/09/07 19:11:12 INFO DAGScheduler: ShuffleMapStage 722 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:12 INFO TaskSchedulerImpl: Removed TaskSet 722.0, whose tasks have all completed, from pool 
16/09/07 19:11:12 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:12 INFO DAGScheduler: running: Set()
16/09/07 19:11:12 INFO DAGScheduler: waiting: Set(ResultStage 723)
16/09/07 19:11:12 INFO DAGScheduler: failed: Set()
16/09/07 19:11:12 INFO DAGScheduler: Missing parents for ResultStage 723: List()
16/09/07 19:11:12 INFO DAGScheduler: Submitting ResultStage 723 (MapPartitionsRDD[2192] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:12 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19405417, maxMem=4042889625
16/09/07 19:11:12 INFO MemoryStore: Block broadcast_1093 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:12 INFO MemoryStore: ensureFreeSpace(2752) called with curMem=19410513, maxMem=4042889625
16/09/07 19:11:12 INFO MemoryStore: Block broadcast_1093_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:12 INFO BlockManagerInfo: Added broadcast_1093_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:12 INFO SparkContext: Created broadcast 1093 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:12 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 723 (MapPartitionsRDD[2192] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:12 INFO TaskSchedulerImpl: Adding task set 723.0 with 8 tasks
16/09/07 19:11:12 INFO TaskSetManager: Starting task 0.0 in stage 723.0 (TID 5784, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 1.0 in stage 723.0 (TID 5785, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 2.0 in stage 723.0 (TID 5786, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 3.0 in stage 723.0 (TID 5787, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 4.0 in stage 723.0 (TID 5788, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 5.0 in stage 723.0 (TID 5789, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 6.0 in stage 723.0 (TID 5790, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:12 INFO TaskSetManager: Starting task 7.0 in stage 723.0 (TID 5791, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:12 INFO Executor: Running task 1.0 in stage 723.0 (TID 5785)
16/09/07 19:11:12 INFO Executor: Running task 2.0 in stage 723.0 (TID 5786)
16/09/07 19:11:12 INFO Executor: Running task 0.0 in stage 723.0 (TID 5784)
16/09/07 19:11:12 INFO Executor: Running task 3.0 in stage 723.0 (TID 5787)
16/09/07 19:11:12 INFO Executor: Running task 5.0 in stage 723.0 (TID 5789)
16/09/07 19:11:12 INFO Executor: Running task 6.0 in stage 723.0 (TID 5790)
16/09/07 19:11:12 INFO Executor: Running task 4.0 in stage 723.0 (TID 5788)
16/09/07 19:11:12 INFO Executor: Running task 7.0 in stage 723.0 (TID 5791)
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:12 INFO Executor: Finished task 2.0 in stage 723.0 (TID 5786). 2160 bytes result sent to driver
16/09/07 19:11:12 INFO Executor: Finished task 4.0 in stage 723.0 (TID 5788). 2160 bytes result sent to driver
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:12 INFO TaskSetManager: Finished task 2.0 in stage 723.0 (TID 5786) in 2 ms on localhost (1/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 4.0 in stage 723.0 (TID 5788) in 2 ms on localhost (2/8)
16/09/07 19:11:12 INFO Executor: Finished task 5.0 in stage 723.0 (TID 5789). 2160 bytes result sent to driver
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:12 INFO Executor: Finished task 3.0 in stage 723.0 (TID 5787). 2160 bytes result sent to driver
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:12 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:12 INFO Executor: Finished task 1.0 in stage 723.0 (TID 5785). 2160 bytes result sent to driver
16/09/07 19:11:12 INFO TaskSetManager: Finished task 5.0 in stage 723.0 (TID 5789) in 6 ms on localhost (3/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 3.0 in stage 723.0 (TID 5787) in 6 ms on localhost (4/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 1.0 in stage 723.0 (TID 5785) in 6 ms on localhost (5/8)
16/09/07 19:11:12 INFO BlockManagerInfo: Removed broadcast_1092_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:12 INFO Executor: Finished task 0.0 in stage 723.0 (TID 5784). 3485524 bytes result sent to driver
16/09/07 19:11:12 INFO Executor: Finished task 6.0 in stage 723.0 (TID 5790). 3485524 bytes result sent to driver
16/09/07 19:11:12 INFO Executor: Finished task 7.0 in stage 723.0 (TID 5791). 3485524 bytes result sent to driver
16/09/07 19:11:12 INFO TaskSetManager: Finished task 0.0 in stage 723.0 (TID 5784) in 896 ms on localhost (6/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 6.0 in stage 723.0 (TID 5790) in 913 ms on localhost (7/8)
16/09/07 19:11:12 INFO TaskSetManager: Finished task 7.0 in stage 723.0 (TID 5791) in 914 ms on localhost (8/8)
16/09/07 19:11:12 INFO TaskSchedulerImpl: Removed TaskSet 723.0, whose tasks have all completed, from pool 
16/09/07 19:11:12 INFO DAGScheduler: ResultStage 723 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.914 s
16/09/07 19:11:12 INFO DAGScheduler: Job 366 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.923346 s
16/09/07 19:11:12 INFO ParameterAveragingTrainingMaster: Completed training of split 45 of 78
16/09/07 19:11:12 INFO ParameterAveragingTrainingMaster: Starting training of split 46 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:13 INFO BlockManagerInfo: Removed broadcast_1093_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:13 INFO ContextCleaner: Cleaned accumulator 724
16/09/07 19:11:13 INFO ContextCleaner: Cleaned accumulator 723
16/09/07 19:11:13 INFO BlockManagerInfo: Removed broadcast_1091_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:13 INFO ContextCleaner: Cleaned shuffle 356
16/09/07 19:11:13 INFO MemoryStore: ensureFreeSpace(18704760) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:13 INFO MemoryStore: Block broadcast_1094 stored as values in memory (estimated size 17.8 MB, free 3.7 GB)
16/09/07 19:11:13 INFO MemoryStore: ensureFreeSpace(168024) called with curMem=19248584, maxMem=4042889625
16/09/07 19:11:13 INFO MemoryStore: Block broadcast_1094_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:13 INFO BlockManagerInfo: Added broadcast_1094_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:13 INFO SparkContext: Created broadcast 1094 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:13 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:13 INFO DAGScheduler: Registering RDD 2193 (repartition at SparkUtils.java:190)
16/09/07 19:11:13 INFO DAGScheduler: Got job 367 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:13 INFO DAGScheduler: Final stage: ResultStage 725(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:13 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 724)
16/09/07 19:11:13 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 724)
16/09/07 19:11:13 INFO DAGScheduler: Submitting ShuffleMapStage 724 (MapPartitionsRDD[2193] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:13 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19416608, maxMem=4042889625
16/09/07 19:11:13 INFO MemoryStore: Block broadcast_1095 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:13 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=19421072, maxMem=4042889625
16/09/07 19:11:13 INFO MemoryStore: Block broadcast_1095_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:13 INFO BlockManagerInfo: Added broadcast_1095_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:13 INFO SparkContext: Created broadcast 1095 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:13 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 724 (MapPartitionsRDD[2193] at repartition at SparkUtils.java:190)
16/09/07 19:11:13 INFO TaskSchedulerImpl: Adding task set 724.0 with 8 tasks
16/09/07 19:11:13 INFO TaskSetManager: Starting task 0.0 in stage 724.0 (TID 5792, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 1.0 in stage 724.0 (TID 5793, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 2.0 in stage 724.0 (TID 5794, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 3.0 in stage 724.0 (TID 5795, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 4.0 in stage 724.0 (TID 5796, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 5.0 in stage 724.0 (TID 5797, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 6.0 in stage 724.0 (TID 5798, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 7.0 in stage 724.0 (TID 5799, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:13 INFO Executor: Running task 2.0 in stage 724.0 (TID 5794)
16/09/07 19:11:13 INFO Executor: Running task 3.0 in stage 724.0 (TID 5795)
16/09/07 19:11:13 INFO Executor: Running task 6.0 in stage 724.0 (TID 5798)
16/09/07 19:11:13 INFO Executor: Running task 0.0 in stage 724.0 (TID 5792)
16/09/07 19:11:13 INFO Executor: Running task 7.0 in stage 724.0 (TID 5799)
16/09/07 19:11:13 INFO Executor: Running task 5.0 in stage 724.0 (TID 5797)
16/09/07 19:11:13 INFO Executor: Running task 4.0 in stage 724.0 (TID 5796)
16/09/07 19:11:13 INFO Executor: Running task 1.0 in stage 724.0 (TID 5793)
16/09/07 19:11:13 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:13 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:13 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:13 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:13 INFO Executor: Finished task 3.0 in stage 724.0 (TID 5795). 2260 bytes result sent to driver
16/09/07 19:11:13 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:13 INFO TaskSetManager: Finished task 3.0 in stage 724.0 (TID 5795) in 2 ms on localhost (1/8)
16/09/07 19:11:13 INFO Executor: Finished task 2.0 in stage 724.0 (TID 5794). 2260 bytes result sent to driver
16/09/07 19:11:13 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:13 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:13 INFO Executor: Finished task 1.0 in stage 724.0 (TID 5793). 2260 bytes result sent to driver
16/09/07 19:11:13 INFO TaskSetManager: Finished task 2.0 in stage 724.0 (TID 5794) in 2 ms on localhost (2/8)
16/09/07 19:11:13 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:13 INFO TaskSetManager: Finished task 1.0 in stage 724.0 (TID 5793) in 3 ms on localhost (3/8)
16/09/07 19:11:13 INFO Executor: Finished task 7.0 in stage 724.0 (TID 5799). 2260 bytes result sent to driver
16/09/07 19:11:13 INFO TaskSetManager: Finished task 7.0 in stage 724.0 (TID 5799) in 3 ms on localhost (4/8)
16/09/07 19:11:13 INFO Executor: Finished task 4.0 in stage 724.0 (TID 5796). 2260 bytes result sent to driver
16/09/07 19:11:13 INFO Executor: Finished task 5.0 in stage 724.0 (TID 5797). 2260 bytes result sent to driver
16/09/07 19:11:13 INFO TaskSetManager: Finished task 4.0 in stage 724.0 (TID 5796) in 3 ms on localhost (5/8)
16/09/07 19:11:13 INFO Executor: Finished task 6.0 in stage 724.0 (TID 5798). 2260 bytes result sent to driver
16/09/07 19:11:13 INFO TaskSetManager: Finished task 5.0 in stage 724.0 (TID 5797) in 3 ms on localhost (6/8)
16/09/07 19:11:13 INFO TaskSetManager: Finished task 6.0 in stage 724.0 (TID 5798) in 3 ms on localhost (7/8)
16/09/07 19:11:13 INFO Executor: Finished task 0.0 in stage 724.0 (TID 5792). 2260 bytes result sent to driver
16/09/07 19:11:13 INFO TaskSetManager: Finished task 0.0 in stage 724.0 (TID 5792) in 3 ms on localhost (8/8)
16/09/07 19:11:13 INFO TaskSchedulerImpl: Removed TaskSet 724.0, whose tasks have all completed, from pool 
16/09/07 19:11:13 INFO DAGScheduler: ShuffleMapStage 724 (repartition at SparkUtils.java:190) finished in 0.003 s
16/09/07 19:11:13 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:13 INFO DAGScheduler: running: Set()
16/09/07 19:11:13 INFO DAGScheduler: waiting: Set(ResultStage 725)
16/09/07 19:11:13 INFO DAGScheduler: failed: Set()
16/09/07 19:11:13 INFO DAGScheduler: Missing parents for ResultStage 725: List()
16/09/07 19:11:13 INFO DAGScheduler: Submitting ResultStage 725 (MapPartitionsRDD[2197] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:13 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19423635, maxMem=4042889625
16/09/07 19:11:13 INFO MemoryStore: Block broadcast_1096 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:13 INFO MemoryStore: ensureFreeSpace(2752) called with curMem=19428731, maxMem=4042889625
16/09/07 19:11:13 INFO MemoryStore: Block broadcast_1096_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:13 INFO BlockManagerInfo: Added broadcast_1096_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:13 INFO SparkContext: Created broadcast 1096 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:13 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 725 (MapPartitionsRDD[2197] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:13 INFO TaskSchedulerImpl: Adding task set 725.0 with 8 tasks
16/09/07 19:11:13 INFO TaskSetManager: Starting task 0.0 in stage 725.0 (TID 5800, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 1.0 in stage 725.0 (TID 5801, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 2.0 in stage 725.0 (TID 5802, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 3.0 in stage 725.0 (TID 5803, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 4.0 in stage 725.0 (TID 5804, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 5.0 in stage 725.0 (TID 5805, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 6.0 in stage 725.0 (TID 5806, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:13 INFO TaskSetManager: Starting task 7.0 in stage 725.0 (TID 5807, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:13 INFO Executor: Running task 1.0 in stage 725.0 (TID 5801)
16/09/07 19:11:13 INFO Executor: Running task 4.0 in stage 725.0 (TID 5804)
16/09/07 19:11:13 INFO Executor: Running task 7.0 in stage 725.0 (TID 5807)
16/09/07 19:11:13 INFO Executor: Running task 6.0 in stage 725.0 (TID 5806)
16/09/07 19:11:13 INFO Executor: Running task 2.0 in stage 725.0 (TID 5802)
16/09/07 19:11:13 INFO Executor: Running task 3.0 in stage 725.0 (TID 5803)
16/09/07 19:11:13 INFO Executor: Running task 0.0 in stage 725.0 (TID 5800)
16/09/07 19:11:13 INFO Executor: Running task 5.0 in stage 725.0 (TID 5805)
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:13 INFO Executor: Finished task 5.0 in stage 725.0 (TID 5805). 2160 bytes result sent to driver
16/09/07 19:11:13 INFO TaskSetManager: Finished task 5.0 in stage 725.0 (TID 5805) in 2 ms on localhost (1/8)
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:13 INFO Executor: Finished task 3.0 in stage 725.0 (TID 5803). 2160 bytes result sent to driver
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:13 INFO TaskSetManager: Finished task 3.0 in stage 725.0 (TID 5803) in 2 ms on localhost (2/8)
16/09/07 19:11:13 INFO Executor: Finished task 2.0 in stage 725.0 (TID 5802). 2160 bytes result sent to driver
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:13 INFO Executor: Finished task 1.0 in stage 725.0 (TID 5801). 2160 bytes result sent to driver
16/09/07 19:11:13 INFO TaskSetManager: Finished task 2.0 in stage 725.0 (TID 5802) in 3 ms on localhost (3/8)
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:13 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:13 INFO TaskSetManager: Finished task 1.0 in stage 725.0 (TID 5801) in 3 ms on localhost (4/8)
16/09/07 19:11:13 INFO Executor: Finished task 4.0 in stage 725.0 (TID 5804). 2160 bytes result sent to driver
16/09/07 19:11:13 INFO TaskSetManager: Finished task 4.0 in stage 725.0 (TID 5804) in 3 ms on localhost (5/8)
16/09/07 19:11:13 INFO BlockManagerInfo: Removed broadcast_1095_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:14 INFO Executor: Finished task 6.0 in stage 725.0 (TID 5806). 3485274 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 6.0 in stage 725.0 (TID 5806) in 777 ms on localhost (6/8)
16/09/07 19:11:14 INFO Executor: Finished task 7.0 in stage 725.0 (TID 5807). 3485274 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 7.0 in stage 725.0 (TID 5807) in 882 ms on localhost (7/8)
16/09/07 19:11:14 INFO Executor: Finished task 0.0 in stage 725.0 (TID 5800). 3485274 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 0.0 in stage 725.0 (TID 5800) in 907 ms on localhost (8/8)
16/09/07 19:11:14 INFO TaskSchedulerImpl: Removed TaskSet 725.0, whose tasks have all completed, from pool 
16/09/07 19:11:14 INFO DAGScheduler: ResultStage 725 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.907 s
16/09/07 19:11:14 INFO DAGScheduler: Job 367 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.914345 s
16/09/07 19:11:14 INFO ParameterAveragingTrainingMaster: Completed training of split 46 of 78
16/09/07 19:11:14 INFO ParameterAveragingTrainingMaster: Starting training of split 47 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:14 INFO BlockManagerInfo: Removed broadcast_1096_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:14 INFO ContextCleaner: Cleaned accumulator 726
16/09/07 19:11:14 INFO ContextCleaner: Cleaned accumulator 725
16/09/07 19:11:14 INFO BlockManagerInfo: Removed broadcast_1094_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:14 INFO ContextCleaner: Cleaned shuffle 357
16/09/07 19:11:14 INFO MemoryStore: ensureFreeSpace(18776896) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:14 INFO MemoryStore: Block broadcast_1097 stored as values in memory (estimated size 17.9 MB, free 3.7 GB)
16/09/07 19:11:14 INFO MemoryStore: ensureFreeSpace(168053) called with curMem=19320720, maxMem=4042889625
16/09/07 19:11:14 INFO MemoryStore: Block broadcast_1097_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:14 INFO BlockManagerInfo: Added broadcast_1097_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:14 INFO SparkContext: Created broadcast 1097 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:14 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:14 INFO DAGScheduler: Registering RDD 2198 (repartition at SparkUtils.java:190)
16/09/07 19:11:14 INFO DAGScheduler: Got job 368 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:14 INFO DAGScheduler: Final stage: ResultStage 727(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:14 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 726)
16/09/07 19:11:14 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 726)
16/09/07 19:11:14 INFO DAGScheduler: Submitting ShuffleMapStage 726 (MapPartitionsRDD[2198] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:14 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19488773, maxMem=4042889625
16/09/07 19:11:14 INFO MemoryStore: Block broadcast_1098 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:14 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19493237, maxMem=4042889625
16/09/07 19:11:14 INFO MemoryStore: Block broadcast_1098_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:14 INFO BlockManagerInfo: Added broadcast_1098_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:14 INFO SparkContext: Created broadcast 1098 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:14 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 726 (MapPartitionsRDD[2198] at repartition at SparkUtils.java:190)
16/09/07 19:11:14 INFO TaskSchedulerImpl: Adding task set 726.0 with 8 tasks
16/09/07 19:11:14 INFO TaskSetManager: Starting task 0.0 in stage 726.0 (TID 5808, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 1.0 in stage 726.0 (TID 5809, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 2.0 in stage 726.0 (TID 5810, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 3.0 in stage 726.0 (TID 5811, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 4.0 in stage 726.0 (TID 5812, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 5.0 in stage 726.0 (TID 5813, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 6.0 in stage 726.0 (TID 5814, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 7.0 in stage 726.0 (TID 5815, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:14 INFO Executor: Running task 4.0 in stage 726.0 (TID 5812)
16/09/07 19:11:14 INFO Executor: Running task 1.0 in stage 726.0 (TID 5809)
16/09/07 19:11:14 INFO Executor: Running task 5.0 in stage 726.0 (TID 5813)
16/09/07 19:11:14 INFO Executor: Running task 7.0 in stage 726.0 (TID 5815)
16/09/07 19:11:14 INFO Executor: Running task 2.0 in stage 726.0 (TID 5810)
16/09/07 19:11:14 INFO Executor: Running task 6.0 in stage 726.0 (TID 5814)
16/09/07 19:11:14 INFO Executor: Running task 3.0 in stage 726.0 (TID 5811)
16/09/07 19:11:14 INFO Executor: Running task 0.0 in stage 726.0 (TID 5808)
16/09/07 19:11:14 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:14 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:14 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:14 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:14 INFO Executor: Finished task 1.0 in stage 726.0 (TID 5809). 2260 bytes result sent to driver
16/09/07 19:11:14 INFO Executor: Finished task 0.0 in stage 726.0 (TID 5808). 2260 bytes result sent to driver
16/09/07 19:11:14 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:14 INFO TaskSetManager: Finished task 1.0 in stage 726.0 (TID 5809) in 2 ms on localhost (1/8)
16/09/07 19:11:14 INFO TaskSetManager: Finished task 0.0 in stage 726.0 (TID 5808) in 2 ms on localhost (2/8)
16/09/07 19:11:14 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:14 INFO Executor: Finished task 4.0 in stage 726.0 (TID 5812). 2260 bytes result sent to driver
16/09/07 19:11:14 INFO Executor: Finished task 7.0 in stage 726.0 (TID 5815). 2260 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 4.0 in stage 726.0 (TID 5812) in 3 ms on localhost (3/8)
16/09/07 19:11:14 INFO TaskSetManager: Finished task 7.0 in stage 726.0 (TID 5815) in 3 ms on localhost (4/8)
16/09/07 19:11:14 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:14 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:14 INFO Executor: Finished task 5.0 in stage 726.0 (TID 5813). 2260 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 5.0 in stage 726.0 (TID 5813) in 4 ms on localhost (5/8)
16/09/07 19:11:14 INFO Executor: Finished task 3.0 in stage 726.0 (TID 5811). 2260 bytes result sent to driver
16/09/07 19:11:14 INFO Executor: Finished task 6.0 in stage 726.0 (TID 5814). 2260 bytes result sent to driver
16/09/07 19:11:14 INFO Executor: Finished task 2.0 in stage 726.0 (TID 5810). 2260 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 3.0 in stage 726.0 (TID 5811) in 4 ms on localhost (6/8)
16/09/07 19:11:14 INFO TaskSetManager: Finished task 6.0 in stage 726.0 (TID 5814) in 4 ms on localhost (7/8)
16/09/07 19:11:14 INFO TaskSetManager: Finished task 2.0 in stage 726.0 (TID 5810) in 4 ms on localhost (8/8)
16/09/07 19:11:14 INFO TaskSchedulerImpl: Removed TaskSet 726.0, whose tasks have all completed, from pool 
16/09/07 19:11:14 INFO DAGScheduler: ShuffleMapStage 726 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:14 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:14 INFO DAGScheduler: running: Set()
16/09/07 19:11:14 INFO DAGScheduler: waiting: Set(ResultStage 727)
16/09/07 19:11:14 INFO DAGScheduler: failed: Set()
16/09/07 19:11:14 INFO DAGScheduler: Missing parents for ResultStage 727: List()
16/09/07 19:11:14 INFO DAGScheduler: Submitting ResultStage 727 (MapPartitionsRDD[2202] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:14 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19495801, maxMem=4042889625
16/09/07 19:11:14 INFO MemoryStore: Block broadcast_1099 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:14 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19500897, maxMem=4042889625
16/09/07 19:11:14 INFO MemoryStore: Block broadcast_1099_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:14 INFO BlockManagerInfo: Added broadcast_1099_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:14 INFO SparkContext: Created broadcast 1099 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:14 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 727 (MapPartitionsRDD[2202] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:14 INFO TaskSchedulerImpl: Adding task set 727.0 with 8 tasks
16/09/07 19:11:14 INFO TaskSetManager: Starting task 0.0 in stage 727.0 (TID 5816, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 1.0 in stage 727.0 (TID 5817, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 2.0 in stage 727.0 (TID 5818, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 3.0 in stage 727.0 (TID 5819, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 4.0 in stage 727.0 (TID 5820, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 5.0 in stage 727.0 (TID 5821, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 6.0 in stage 727.0 (TID 5822, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:14 INFO TaskSetManager: Starting task 7.0 in stage 727.0 (TID 5823, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:14 INFO Executor: Running task 2.0 in stage 727.0 (TID 5818)
16/09/07 19:11:14 INFO Executor: Running task 4.0 in stage 727.0 (TID 5820)
16/09/07 19:11:14 INFO Executor: Running task 1.0 in stage 727.0 (TID 5817)
16/09/07 19:11:14 INFO Executor: Running task 6.0 in stage 727.0 (TID 5822)
16/09/07 19:11:14 INFO Executor: Running task 5.0 in stage 727.0 (TID 5821)
16/09/07 19:11:14 INFO Executor: Running task 0.0 in stage 727.0 (TID 5816)
16/09/07 19:11:14 INFO Executor: Running task 3.0 in stage 727.0 (TID 5819)
16/09/07 19:11:14 INFO Executor: Running task 7.0 in stage 727.0 (TID 5823)
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:14 INFO Executor: Finished task 5.0 in stage 727.0 (TID 5821). 2160 bytes result sent to driver
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:14 INFO Executor: Finished task 3.0 in stage 727.0 (TID 5819). 2160 bytes result sent to driver
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:14 INFO TaskSetManager: Finished task 3.0 in stage 727.0 (TID 5819) in 1 ms on localhost (1/8)
16/09/07 19:11:14 INFO TaskSetManager: Finished task 5.0 in stage 727.0 (TID 5821) in 1 ms on localhost (2/8)
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:14 INFO Executor: Finished task 2.0 in stage 727.0 (TID 5818). 2160 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 2.0 in stage 727.0 (TID 5818) in 2 ms on localhost (3/8)
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:14 INFO Executor: Finished task 1.0 in stage 727.0 (TID 5817). 2160 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 1.0 in stage 727.0 (TID 5817) in 3 ms on localhost (4/8)
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:14 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:14 INFO Executor: Finished task 4.0 in stage 727.0 (TID 5820). 2160 bytes result sent to driver
16/09/07 19:11:14 INFO TaskSetManager: Finished task 4.0 in stage 727.0 (TID 5820) in 4 ms on localhost (5/8)
16/09/07 19:11:14 INFO BlockManagerInfo: Removed broadcast_1098_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:15 INFO Executor: Finished task 7.0 in stage 727.0 (TID 5823). 3485274 bytes result sent to driver
16/09/07 19:11:15 INFO Executor: Finished task 0.0 in stage 727.0 (TID 5816). 3485274 bytes result sent to driver
16/09/07 19:11:15 INFO Executor: Finished task 6.0 in stage 727.0 (TID 5822). 3485274 bytes result sent to driver
16/09/07 19:11:15 INFO TaskSetManager: Finished task 6.0 in stage 727.0 (TID 5822) in 919 ms on localhost (6/8)
16/09/07 19:11:15 INFO TaskSetManager: Finished task 7.0 in stage 727.0 (TID 5823) in 930 ms on localhost (7/8)
16/09/07 19:11:15 INFO TaskSetManager: Finished task 0.0 in stage 727.0 (TID 5816) in 933 ms on localhost (8/8)
16/09/07 19:11:15 INFO TaskSchedulerImpl: Removed TaskSet 727.0, whose tasks have all completed, from pool 
16/09/07 19:11:15 INFO DAGScheduler: ResultStage 727 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.933 s
16/09/07 19:11:15 INFO DAGScheduler: Job 368 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.941076 s
16/09/07 19:11:15 INFO ParameterAveragingTrainingMaster: Completed training of split 47 of 78
16/09/07 19:11:15 INFO ParameterAveragingTrainingMaster: Starting training of split 48 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:15 INFO MemoryStore: ensureFreeSpace(18815744) called with curMem=19496612, maxMem=4042889625
16/09/07 19:11:15 INFO MemoryStore: Block broadcast_1100 stored as values in memory (estimated size 17.9 MB, free 3.7 GB)
16/09/07 19:11:15 INFO MemoryStore: ensureFreeSpace(168030) called with curMem=38312356, maxMem=4042889625
16/09/07 19:11:15 INFO MemoryStore: Block broadcast_1100_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:15 INFO BlockManagerInfo: Added broadcast_1100_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:15 INFO SparkContext: Created broadcast 1100 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:15 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:15 INFO DAGScheduler: Registering RDD 2203 (repartition at SparkUtils.java:190)
16/09/07 19:11:15 INFO DAGScheduler: Got job 369 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:15 INFO DAGScheduler: Final stage: ResultStage 729(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:15 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 728)
16/09/07 19:11:15 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 728)
16/09/07 19:11:15 INFO DAGScheduler: Submitting ShuffleMapStage 728 (MapPartitionsRDD[2203] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:15 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=38480386, maxMem=4042889625
16/09/07 19:11:15 INFO MemoryStore: Block broadcast_1101 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:15 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=38484850, maxMem=4042889625
16/09/07 19:11:15 INFO MemoryStore: Block broadcast_1101_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:15 INFO BlockManagerInfo: Added broadcast_1101_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:15 INFO SparkContext: Created broadcast 1101 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:15 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 728 (MapPartitionsRDD[2203] at repartition at SparkUtils.java:190)
16/09/07 19:11:15 INFO TaskSchedulerImpl: Adding task set 728.0 with 8 tasks
16/09/07 19:11:15 INFO TaskSetManager: Starting task 0.0 in stage 728.0 (TID 5824, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 1.0 in stage 728.0 (TID 5825, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 2.0 in stage 728.0 (TID 5826, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 3.0 in stage 728.0 (TID 5827, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 4.0 in stage 728.0 (TID 5828, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 5.0 in stage 728.0 (TID 5829, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 6.0 in stage 728.0 (TID 5830, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 7.0 in stage 728.0 (TID 5831, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:15 INFO Executor: Running task 0.0 in stage 728.0 (TID 5824)
16/09/07 19:11:15 INFO Executor: Running task 3.0 in stage 728.0 (TID 5827)
16/09/07 19:11:15 INFO Executor: Running task 4.0 in stage 728.0 (TID 5828)
16/09/07 19:11:15 INFO Executor: Running task 2.0 in stage 728.0 (TID 5826)
16/09/07 19:11:15 INFO Executor: Running task 1.0 in stage 728.0 (TID 5825)
16/09/07 19:11:15 INFO Executor: Running task 5.0 in stage 728.0 (TID 5829)
16/09/07 19:11:15 INFO Executor: Running task 7.0 in stage 728.0 (TID 5831)
16/09/07 19:11:15 INFO Executor: Running task 6.0 in stage 728.0 (TID 5830)
16/09/07 19:11:15 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:15 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:15 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:15 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:15 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:15 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:15 INFO BlockManagerInfo: Removed broadcast_1099_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:15 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:15 INFO ContextCleaner: Cleaned accumulator 728
16/09/07 19:11:15 INFO ContextCleaner: Cleaned accumulator 727
16/09/07 19:11:15 INFO BlockManagerInfo: Removed broadcast_1097_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:15 INFO ContextCleaner: Cleaned shuffle 358
16/09/07 19:11:15 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:15 INFO Executor: Finished task 0.0 in stage 728.0 (TID 5824). 2260 bytes result sent to driver
16/09/07 19:11:15 INFO TaskSetManager: Finished task 0.0 in stage 728.0 (TID 5824) in 92 ms on localhost (1/8)
16/09/07 19:11:15 INFO Executor: Finished task 3.0 in stage 728.0 (TID 5827). 2260 bytes result sent to driver
16/09/07 19:11:15 INFO Executor: Finished task 1.0 in stage 728.0 (TID 5825). 2260 bytes result sent to driver
16/09/07 19:11:15 INFO Executor: Finished task 7.0 in stage 728.0 (TID 5831). 2260 bytes result sent to driver
16/09/07 19:11:15 INFO TaskSetManager: Finished task 1.0 in stage 728.0 (TID 5825) in 93 ms on localhost (2/8)
16/09/07 19:11:15 INFO TaskSetManager: Finished task 3.0 in stage 728.0 (TID 5827) in 93 ms on localhost (3/8)
16/09/07 19:11:15 INFO TaskSetManager: Finished task 7.0 in stage 728.0 (TID 5831) in 93 ms on localhost (4/8)
16/09/07 19:11:15 INFO Executor: Finished task 6.0 in stage 728.0 (TID 5830). 2260 bytes result sent to driver
16/09/07 19:11:15 INFO Executor: Finished task 4.0 in stage 728.0 (TID 5828). 2260 bytes result sent to driver
16/09/07 19:11:15 INFO Executor: Finished task 5.0 in stage 728.0 (TID 5829). 2260 bytes result sent to driver
16/09/07 19:11:15 INFO Executor: Finished task 2.0 in stage 728.0 (TID 5826). 2260 bytes result sent to driver
16/09/07 19:11:15 INFO TaskSetManager: Finished task 6.0 in stage 728.0 (TID 5830) in 93 ms on localhost (5/8)
16/09/07 19:11:15 INFO TaskSetManager: Finished task 5.0 in stage 728.0 (TID 5829) in 93 ms on localhost (6/8)
16/09/07 19:11:15 INFO TaskSetManager: Finished task 4.0 in stage 728.0 (TID 5828) in 93 ms on localhost (7/8)
16/09/07 19:11:15 INFO TaskSetManager: Finished task 2.0 in stage 728.0 (TID 5826) in 93 ms on localhost (8/8)
16/09/07 19:11:15 INFO DAGScheduler: ShuffleMapStage 728 (repartition at SparkUtils.java:190) finished in 0.093 s
16/09/07 19:11:15 INFO TaskSchedulerImpl: Removed TaskSet 728.0, whose tasks have all completed, from pool 
16/09/07 19:11:15 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:15 INFO DAGScheduler: running: Set()
16/09/07 19:11:15 INFO DAGScheduler: waiting: Set(ResultStage 729)
16/09/07 19:11:15 INFO DAGScheduler: failed: Set()
16/09/07 19:11:15 INFO DAGScheduler: Missing parents for ResultStage 729: List()
16/09/07 19:11:15 INFO DAGScheduler: Submitting ResultStage 729 (MapPartitionsRDD[2207] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:15 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19534624, maxMem=4042889625
16/09/07 19:11:15 INFO MemoryStore: Block broadcast_1102 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:15 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=19539720, maxMem=4042889625
16/09/07 19:11:15 INFO MemoryStore: Block broadcast_1102_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:15 INFO BlockManagerInfo: Added broadcast_1102_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:15 INFO SparkContext: Created broadcast 1102 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:15 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 729 (MapPartitionsRDD[2207] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:15 INFO TaskSchedulerImpl: Adding task set 729.0 with 8 tasks
16/09/07 19:11:15 INFO TaskSetManager: Starting task 0.0 in stage 729.0 (TID 5832, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 1.0 in stage 729.0 (TID 5833, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 2.0 in stage 729.0 (TID 5834, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 3.0 in stage 729.0 (TID 5835, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 4.0 in stage 729.0 (TID 5836, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 5.0 in stage 729.0 (TID 5837, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 6.0 in stage 729.0 (TID 5838, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:15 INFO TaskSetManager: Starting task 7.0 in stage 729.0 (TID 5839, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:15 INFO Executor: Running task 1.0 in stage 729.0 (TID 5833)
16/09/07 19:11:15 INFO Executor: Running task 3.0 in stage 729.0 (TID 5835)
16/09/07 19:11:15 INFO Executor: Running task 6.0 in stage 729.0 (TID 5838)
16/09/07 19:11:15 INFO Executor: Running task 0.0 in stage 729.0 (TID 5832)
16/09/07 19:11:15 INFO Executor: Running task 2.0 in stage 729.0 (TID 5834)
16/09/07 19:11:15 INFO Executor: Running task 4.0 in stage 729.0 (TID 5836)
16/09/07 19:11:15 INFO Executor: Running task 7.0 in stage 729.0 (TID 5839)
16/09/07 19:11:15 INFO Executor: Running task 5.0 in stage 729.0 (TID 5837)
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:15 INFO Executor: Finished task 1.0 in stage 729.0 (TID 5833). 2160 bytes result sent to driver
16/09/07 19:11:15 INFO Executor: Finished task 5.0 in stage 729.0 (TID 5837). 2160 bytes result sent to driver
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:15 INFO TaskSetManager: Finished task 1.0 in stage 729.0 (TID 5833) in 1 ms on localhost (1/8)
16/09/07 19:11:15 INFO TaskSetManager: Finished task 5.0 in stage 729.0 (TID 5837) in 1 ms on localhost (2/8)
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:15 INFO Executor: Finished task 2.0 in stage 729.0 (TID 5834). 2160 bytes result sent to driver
16/09/07 19:11:15 INFO TaskSetManager: Finished task 2.0 in stage 729.0 (TID 5834) in 2 ms on localhost (3/8)
16/09/07 19:11:15 INFO Executor: Finished task 4.0 in stage 729.0 (TID 5836). 2160 bytes result sent to driver
16/09/07 19:11:15 INFO TaskSetManager: Finished task 4.0 in stage 729.0 (TID 5836) in 2 ms on localhost (4/8)
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:15 INFO Executor: Finished task 3.0 in stage 729.0 (TID 5835). 2160 bytes result sent to driver
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:15 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:15 INFO TaskSetManager: Finished task 3.0 in stage 729.0 (TID 5835) in 2 ms on localhost (5/8)
16/09/07 19:11:15 INFO BlockManagerInfo: Removed broadcast_1101_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:16 INFO Executor: Finished task 7.0 in stage 729.0 (TID 5839). 3485274 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 7.0 in stage 729.0 (TID 5839) in 764 ms on localhost (6/8)
16/09/07 19:11:16 INFO Executor: Finished task 6.0 in stage 729.0 (TID 5838). 3485274 bytes result sent to driver
16/09/07 19:11:16 INFO Executor: Finished task 0.0 in stage 729.0 (TID 5832). 3485274 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 0.0 in stage 729.0 (TID 5832) in 831 ms on localhost (7/8)
16/09/07 19:11:16 INFO TaskSetManager: Finished task 6.0 in stage 729.0 (TID 5838) in 830 ms on localhost (8/8)
16/09/07 19:11:16 INFO TaskSchedulerImpl: Removed TaskSet 729.0, whose tasks have all completed, from pool 
16/09/07 19:11:16 INFO DAGScheduler: ResultStage 729 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.831 s
16/09/07 19:11:16 INFO DAGScheduler: Job 369 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.926845 s
16/09/07 19:11:16 INFO ParameterAveragingTrainingMaster: Completed training of split 48 of 78
16/09/07 19:11:16 INFO ParameterAveragingTrainingMaster: Starting training of split 49 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:16 INFO BlockManagerInfo: Removed broadcast_1102_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:16 INFO ContextCleaner: Cleaned accumulator 730
16/09/07 19:11:16 INFO ContextCleaner: Cleaned accumulator 729
16/09/07 19:11:16 INFO BlockManagerInfo: Removed broadcast_1100_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:16 INFO ContextCleaner: Cleaned shuffle 359
16/09/07 19:11:16 INFO MemoryStore: ensureFreeSpace(18651472) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:16 INFO MemoryStore: Block broadcast_1103 stored as values in memory (estimated size 17.8 MB, free 3.7 GB)
16/09/07 19:11:16 INFO MemoryStore: ensureFreeSpace(168003) called with curMem=19195296, maxMem=4042889625
16/09/07 19:11:16 INFO MemoryStore: Block broadcast_1103_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:16 INFO BlockManagerInfo: Added broadcast_1103_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:16 INFO SparkContext: Created broadcast 1103 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:16 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:16 INFO DAGScheduler: Registering RDD 2208 (repartition at SparkUtils.java:190)
16/09/07 19:11:16 INFO DAGScheduler: Got job 370 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:16 INFO DAGScheduler: Final stage: ResultStage 731(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:16 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 730)
16/09/07 19:11:16 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 730)
16/09/07 19:11:16 INFO DAGScheduler: Submitting ShuffleMapStage 730 (MapPartitionsRDD[2208] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:16 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19363299, maxMem=4042889625
16/09/07 19:11:16 INFO MemoryStore: Block broadcast_1104 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:16 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19367763, maxMem=4042889625
16/09/07 19:11:16 INFO MemoryStore: Block broadcast_1104_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:16 INFO BlockManagerInfo: Added broadcast_1104_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:16 INFO SparkContext: Created broadcast 1104 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:16 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 730 (MapPartitionsRDD[2208] at repartition at SparkUtils.java:190)
16/09/07 19:11:16 INFO TaskSchedulerImpl: Adding task set 730.0 with 8 tasks
16/09/07 19:11:16 INFO TaskSetManager: Starting task 0.0 in stage 730.0 (TID 5840, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 1.0 in stage 730.0 (TID 5841, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 2.0 in stage 730.0 (TID 5842, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 3.0 in stage 730.0 (TID 5843, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 4.0 in stage 730.0 (TID 5844, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 5.0 in stage 730.0 (TID 5845, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 6.0 in stage 730.0 (TID 5846, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 7.0 in stage 730.0 (TID 5847, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:16 INFO Executor: Running task 1.0 in stage 730.0 (TID 5841)
16/09/07 19:11:16 INFO Executor: Running task 4.0 in stage 730.0 (TID 5844)
16/09/07 19:11:16 INFO Executor: Running task 3.0 in stage 730.0 (TID 5843)
16/09/07 19:11:16 INFO Executor: Running task 0.0 in stage 730.0 (TID 5840)
16/09/07 19:11:16 INFO Executor: Running task 6.0 in stage 730.0 (TID 5846)
16/09/07 19:11:16 INFO Executor: Running task 2.0 in stage 730.0 (TID 5842)
16/09/07 19:11:16 INFO Executor: Running task 5.0 in stage 730.0 (TID 5845)
16/09/07 19:11:16 INFO Executor: Running task 7.0 in stage 730.0 (TID 5847)
16/09/07 19:11:16 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:16 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:16 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:16 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:16 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:16 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:16 INFO Executor: Finished task 4.0 in stage 730.0 (TID 5844). 2260 bytes result sent to driver
16/09/07 19:11:16 INFO Executor: Finished task 1.0 in stage 730.0 (TID 5841). 2260 bytes result sent to driver
16/09/07 19:11:16 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:16 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:16 INFO Executor: Finished task 2.0 in stage 730.0 (TID 5842). 2260 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 4.0 in stage 730.0 (TID 5844) in 3 ms on localhost (1/8)
16/09/07 19:11:16 INFO TaskSetManager: Finished task 1.0 in stage 730.0 (TID 5841) in 3 ms on localhost (2/8)
16/09/07 19:11:16 INFO Executor: Finished task 7.0 in stage 730.0 (TID 5847). 2260 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 2.0 in stage 730.0 (TID 5842) in 3 ms on localhost (3/8)
16/09/07 19:11:16 INFO TaskSetManager: Finished task 7.0 in stage 730.0 (TID 5847) in 2 ms on localhost (4/8)
16/09/07 19:11:16 INFO Executor: Finished task 3.0 in stage 730.0 (TID 5843). 2260 bytes result sent to driver
16/09/07 19:11:16 INFO Executor: Finished task 5.0 in stage 730.0 (TID 5845). 2260 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 3.0 in stage 730.0 (TID 5843) in 4 ms on localhost (5/8)
16/09/07 19:11:16 INFO TaskSetManager: Finished task 5.0 in stage 730.0 (TID 5845) in 3 ms on localhost (6/8)
16/09/07 19:11:16 INFO Executor: Finished task 6.0 in stage 730.0 (TID 5846). 2260 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 6.0 in stage 730.0 (TID 5846) in 3 ms on localhost (7/8)
16/09/07 19:11:16 INFO Executor: Finished task 0.0 in stage 730.0 (TID 5840). 2260 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 0.0 in stage 730.0 (TID 5840) in 4 ms on localhost (8/8)
16/09/07 19:11:16 INFO TaskSchedulerImpl: Removed TaskSet 730.0, whose tasks have all completed, from pool 
16/09/07 19:11:16 INFO DAGScheduler: ShuffleMapStage 730 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:16 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:16 INFO DAGScheduler: running: Set()
16/09/07 19:11:16 INFO DAGScheduler: waiting: Set(ResultStage 731)
16/09/07 19:11:16 INFO DAGScheduler: failed: Set()
16/09/07 19:11:16 INFO DAGScheduler: Missing parents for ResultStage 731: List()
16/09/07 19:11:16 INFO DAGScheduler: Submitting ResultStage 731 (MapPartitionsRDD[2212] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:16 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19370327, maxMem=4042889625
16/09/07 19:11:16 INFO MemoryStore: Block broadcast_1105 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:16 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19375423, maxMem=4042889625
16/09/07 19:11:16 INFO MemoryStore: Block broadcast_1105_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:16 INFO BlockManagerInfo: Added broadcast_1105_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:16 INFO SparkContext: Created broadcast 1105 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:16 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 731 (MapPartitionsRDD[2212] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:16 INFO TaskSchedulerImpl: Adding task set 731.0 with 8 tasks
16/09/07 19:11:16 INFO TaskSetManager: Starting task 0.0 in stage 731.0 (TID 5848, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 1.0 in stage 731.0 (TID 5849, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 2.0 in stage 731.0 (TID 5850, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 3.0 in stage 731.0 (TID 5851, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 4.0 in stage 731.0 (TID 5852, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 5.0 in stage 731.0 (TID 5853, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 6.0 in stage 731.0 (TID 5854, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:16 INFO TaskSetManager: Starting task 7.0 in stage 731.0 (TID 5855, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:16 INFO Executor: Running task 1.0 in stage 731.0 (TID 5849)
16/09/07 19:11:16 INFO Executor: Running task 2.0 in stage 731.0 (TID 5850)
16/09/07 19:11:16 INFO Executor: Running task 3.0 in stage 731.0 (TID 5851)
16/09/07 19:11:16 INFO Executor: Running task 0.0 in stage 731.0 (TID 5848)
16/09/07 19:11:16 INFO Executor: Running task 6.0 in stage 731.0 (TID 5854)
16/09/07 19:11:16 INFO Executor: Running task 5.0 in stage 731.0 (TID 5853)
16/09/07 19:11:16 INFO Executor: Running task 7.0 in stage 731.0 (TID 5855)
16/09/07 19:11:16 INFO Executor: Running task 4.0 in stage 731.0 (TID 5852)
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:16 INFO Executor: Finished task 4.0 in stage 731.0 (TID 5852). 2160 bytes result sent to driver
16/09/07 19:11:16 INFO Executor: Finished task 3.0 in stage 731.0 (TID 5851). 2160 bytes result sent to driver
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:16 INFO TaskSetManager: Finished task 4.0 in stage 731.0 (TID 5852) in 1 ms on localhost (1/8)
16/09/07 19:11:16 INFO Executor: Finished task 5.0 in stage 731.0 (TID 5853). 2160 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 3.0 in stage 731.0 (TID 5851) in 2 ms on localhost (2/8)
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:16 INFO TaskSetManager: Finished task 5.0 in stage 731.0 (TID 5853) in 2 ms on localhost (3/8)
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:16 INFO Executor: Finished task 2.0 in stage 731.0 (TID 5850). 2160 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 2.0 in stage 731.0 (TID 5850) in 3 ms on localhost (4/8)
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:16 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:16 INFO Executor: Finished task 1.0 in stage 731.0 (TID 5849). 2160 bytes result sent to driver
16/09/07 19:11:16 INFO TaskSetManager: Finished task 1.0 in stage 731.0 (TID 5849) in 4 ms on localhost (5/8)
16/09/07 19:11:17 INFO BlockManagerInfo: Removed broadcast_1104_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:17 INFO Executor: Finished task 0.0 in stage 731.0 (TID 5848). 3485274 bytes result sent to driver
16/09/07 19:11:17 INFO Executor: Finished task 7.0 in stage 731.0 (TID 5855). 3485274 bytes result sent to driver
16/09/07 19:11:17 INFO Executor: Finished task 6.0 in stage 731.0 (TID 5854). 3485274 bytes result sent to driver
16/09/07 19:11:17 INFO TaskSetManager: Finished task 6.0 in stage 731.0 (TID 5854) in 893 ms on localhost (6/8)
16/09/07 19:11:17 INFO TaskSetManager: Finished task 7.0 in stage 731.0 (TID 5855) in 905 ms on localhost (7/8)
16/09/07 19:11:17 INFO TaskSetManager: Finished task 0.0 in stage 731.0 (TID 5848) in 906 ms on localhost (8/8)
16/09/07 19:11:17 INFO TaskSchedulerImpl: Removed TaskSet 731.0, whose tasks have all completed, from pool 
16/09/07 19:11:17 INFO DAGScheduler: ResultStage 731 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.906 s
16/09/07 19:11:17 INFO DAGScheduler: Job 370 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.914311 s
16/09/07 19:11:17 INFO ParameterAveragingTrainingMaster: Completed training of split 49 of 78
16/09/07 19:11:17 INFO ParameterAveragingTrainingMaster: Starting training of split 50 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:18 INFO ContextCleaner: Cleaned accumulator 731
16/09/07 19:11:18 INFO BlockManagerInfo: Removed broadcast_1105_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:18 INFO ContextCleaner: Cleaned accumulator 732
16/09/07 19:11:18 INFO BlockManagerInfo: Removed broadcast_1103_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:18 INFO ContextCleaner: Cleaned shuffle 360
16/09/07 19:11:18 INFO MemoryStore: ensureFreeSpace(18840056) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:18 INFO MemoryStore: Block broadcast_1106 stored as values in memory (estimated size 18.0 MB, free 3.7 GB)
16/09/07 19:11:18 INFO MemoryStore: ensureFreeSpace(167976) called with curMem=19383880, maxMem=4042889625
16/09/07 19:11:18 INFO MemoryStore: Block broadcast_1106_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:18 INFO BlockManagerInfo: Added broadcast_1106_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:18 INFO SparkContext: Created broadcast 1106 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:18 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:18 INFO DAGScheduler: Registering RDD 2213 (repartition at SparkUtils.java:190)
16/09/07 19:11:18 INFO DAGScheduler: Got job 371 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:18 INFO DAGScheduler: Final stage: ResultStage 733(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:18 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 732)
16/09/07 19:11:18 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 732)
16/09/07 19:11:18 INFO DAGScheduler: Submitting ShuffleMapStage 732 (MapPartitionsRDD[2213] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:18 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19551856, maxMem=4042889625
16/09/07 19:11:18 INFO MemoryStore: Block broadcast_1107 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:18 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19556320, maxMem=4042889625
16/09/07 19:11:18 INFO MemoryStore: Block broadcast_1107_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:18 INFO BlockManagerInfo: Added broadcast_1107_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:18 INFO SparkContext: Created broadcast 1107 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:18 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 732 (MapPartitionsRDD[2213] at repartition at SparkUtils.java:190)
16/09/07 19:11:18 INFO TaskSchedulerImpl: Adding task set 732.0 with 8 tasks
16/09/07 19:11:18 INFO TaskSetManager: Starting task 0.0 in stage 732.0 (TID 5856, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 1.0 in stage 732.0 (TID 5857, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 2.0 in stage 732.0 (TID 5858, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 3.0 in stage 732.0 (TID 5859, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 4.0 in stage 732.0 (TID 5860, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 5.0 in stage 732.0 (TID 5861, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 6.0 in stage 732.0 (TID 5862, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 7.0 in stage 732.0 (TID 5863, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:18 INFO Executor: Running task 1.0 in stage 732.0 (TID 5857)
16/09/07 19:11:18 INFO Executor: Running task 5.0 in stage 732.0 (TID 5861)
16/09/07 19:11:18 INFO Executor: Running task 7.0 in stage 732.0 (TID 5863)
16/09/07 19:11:18 INFO Executor: Running task 4.0 in stage 732.0 (TID 5860)
16/09/07 19:11:18 INFO Executor: Running task 2.0 in stage 732.0 (TID 5858)
16/09/07 19:11:18 INFO Executor: Running task 0.0 in stage 732.0 (TID 5856)
16/09/07 19:11:18 INFO Executor: Running task 6.0 in stage 732.0 (TID 5862)
16/09/07 19:11:18 INFO Executor: Running task 3.0 in stage 732.0 (TID 5859)
16/09/07 19:11:18 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:18 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:18 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:18 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:18 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:18 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:18 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:18 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:18 INFO Executor: Finished task 1.0 in stage 732.0 (TID 5857). 2260 bytes result sent to driver
16/09/07 19:11:18 INFO Executor: Finished task 6.0 in stage 732.0 (TID 5862). 2260 bytes result sent to driver
16/09/07 19:11:18 INFO Executor: Finished task 0.0 in stage 732.0 (TID 5856). 2260 bytes result sent to driver
16/09/07 19:11:18 INFO Executor: Finished task 3.0 in stage 732.0 (TID 5859). 2260 bytes result sent to driver
16/09/07 19:11:18 INFO TaskSetManager: Finished task 1.0 in stage 732.0 (TID 5857) in 3 ms on localhost (1/8)
16/09/07 19:11:18 INFO TaskSetManager: Finished task 6.0 in stage 732.0 (TID 5862) in 2 ms on localhost (2/8)
16/09/07 19:11:18 INFO TaskSetManager: Finished task 0.0 in stage 732.0 (TID 5856) in 3 ms on localhost (3/8)
16/09/07 19:11:18 INFO TaskSetManager: Finished task 3.0 in stage 732.0 (TID 5859) in 2 ms on localhost (4/8)
16/09/07 19:11:18 INFO Executor: Finished task 5.0 in stage 732.0 (TID 5861). 2260 bytes result sent to driver
16/09/07 19:11:18 INFO Executor: Finished task 7.0 in stage 732.0 (TID 5863). 2260 bytes result sent to driver
16/09/07 19:11:18 INFO Executor: Finished task 4.0 in stage 732.0 (TID 5860). 2260 bytes result sent to driver
16/09/07 19:11:18 INFO Executor: Finished task 2.0 in stage 732.0 (TID 5858). 2260 bytes result sent to driver
16/09/07 19:11:18 INFO TaskSetManager: Finished task 7.0 in stage 732.0 (TID 5863) in 3 ms on localhost (5/8)
16/09/07 19:11:18 INFO TaskSetManager: Finished task 2.0 in stage 732.0 (TID 5858) in 4 ms on localhost (6/8)
16/09/07 19:11:18 INFO TaskSetManager: Finished task 5.0 in stage 732.0 (TID 5861) in 3 ms on localhost (7/8)
16/09/07 19:11:18 INFO TaskSetManager: Finished task 4.0 in stage 732.0 (TID 5860) in 3 ms on localhost (8/8)
16/09/07 19:11:18 INFO TaskSchedulerImpl: Removed TaskSet 732.0, whose tasks have all completed, from pool 
16/09/07 19:11:18 INFO DAGScheduler: ShuffleMapStage 732 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:18 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:18 INFO DAGScheduler: running: Set()
16/09/07 19:11:18 INFO DAGScheduler: waiting: Set(ResultStage 733)
16/09/07 19:11:18 INFO DAGScheduler: failed: Set()
16/09/07 19:11:18 INFO DAGScheduler: Missing parents for ResultStage 733: List()
16/09/07 19:11:18 INFO DAGScheduler: Submitting ResultStage 733 (MapPartitionsRDD[2217] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:18 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19558884, maxMem=4042889625
16/09/07 19:11:18 INFO MemoryStore: Block broadcast_1108 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:18 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19563980, maxMem=4042889625
16/09/07 19:11:18 INFO MemoryStore: Block broadcast_1108_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:18 INFO BlockManagerInfo: Added broadcast_1108_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:18 INFO SparkContext: Created broadcast 1108 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:18 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 733 (MapPartitionsRDD[2217] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:18 INFO TaskSchedulerImpl: Adding task set 733.0 with 8 tasks
16/09/07 19:11:18 INFO TaskSetManager: Starting task 0.0 in stage 733.0 (TID 5864, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 1.0 in stage 733.0 (TID 5865, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 2.0 in stage 733.0 (TID 5866, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 3.0 in stage 733.0 (TID 5867, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 4.0 in stage 733.0 (TID 5868, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 5.0 in stage 733.0 (TID 5869, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 6.0 in stage 733.0 (TID 5870, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:18 INFO TaskSetManager: Starting task 7.0 in stage 733.0 (TID 5871, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:18 INFO Executor: Running task 1.0 in stage 733.0 (TID 5865)
16/09/07 19:11:18 INFO Executor: Running task 4.0 in stage 733.0 (TID 5868)
16/09/07 19:11:18 INFO Executor: Running task 5.0 in stage 733.0 (TID 5869)
16/09/07 19:11:18 INFO Executor: Running task 3.0 in stage 733.0 (TID 5867)
16/09/07 19:11:18 INFO Executor: Running task 2.0 in stage 733.0 (TID 5866)
16/09/07 19:11:18 INFO Executor: Running task 0.0 in stage 733.0 (TID 5864)
16/09/07 19:11:18 INFO Executor: Running task 7.0 in stage 733.0 (TID 5871)
16/09/07 19:11:18 INFO Executor: Running task 6.0 in stage 733.0 (TID 5870)
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:18 INFO Executor: Finished task 4.0 in stage 733.0 (TID 5868). 2160 bytes result sent to driver
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:18 INFO TaskSetManager: Finished task 4.0 in stage 733.0 (TID 5868) in 2 ms on localhost (1/8)
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:18 INFO Executor: Finished task 3.0 in stage 733.0 (TID 5867). 2160 bytes result sent to driver
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:18 INFO Executor: Finished task 2.0 in stage 733.0 (TID 5866). 2160 bytes result sent to driver
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:18 INFO TaskSetManager: Finished task 3.0 in stage 733.0 (TID 5867) in 2 ms on localhost (2/8)
16/09/07 19:11:18 INFO TaskSetManager: Finished task 2.0 in stage 733.0 (TID 5866) in 3 ms on localhost (3/8)
16/09/07 19:11:18 INFO Executor: Finished task 5.0 in stage 733.0 (TID 5869). 2160 bytes result sent to driver
16/09/07 19:11:18 INFO TaskSetManager: Finished task 5.0 in stage 733.0 (TID 5869) in 3 ms on localhost (4/8)
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:18 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:18 INFO Executor: Finished task 1.0 in stage 733.0 (TID 5865). 2160 bytes result sent to driver
16/09/07 19:11:18 INFO TaskSetManager: Finished task 1.0 in stage 733.0 (TID 5865) in 4 ms on localhost (5/8)
16/09/07 19:11:18 INFO BlockManagerInfo: Removed broadcast_1107_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:19 INFO Executor: Finished task 6.0 in stage 733.0 (TID 5870). 3485274 bytes result sent to driver
16/09/07 19:11:19 INFO Executor: Finished task 0.0 in stage 733.0 (TID 5864). 3485274 bytes result sent to driver
16/09/07 19:11:19 INFO Executor: Finished task 7.0 in stage 733.0 (TID 5871). 3485274 bytes result sent to driver
16/09/07 19:11:19 INFO TaskSetManager: Finished task 7.0 in stage 733.0 (TID 5871) in 924 ms on localhost (6/8)
16/09/07 19:11:19 INFO TaskSetManager: Finished task 6.0 in stage 733.0 (TID 5870) in 935 ms on localhost (7/8)
16/09/07 19:11:19 INFO TaskSetManager: Finished task 0.0 in stage 733.0 (TID 5864) in 935 ms on localhost (8/8)
16/09/07 19:11:19 INFO TaskSchedulerImpl: Removed TaskSet 733.0, whose tasks have all completed, from pool 
16/09/07 19:11:19 INFO DAGScheduler: ResultStage 733 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.935 s
16/09/07 19:11:19 INFO DAGScheduler: Job 371 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.942422 s
16/09/07 19:11:19 INFO ParameterAveragingTrainingMaster: Completed training of split 50 of 78
16/09/07 19:11:19 INFO ParameterAveragingTrainingMaster: Starting training of split 51 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:19 INFO BlockManagerInfo: Removed broadcast_1108_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:19 INFO ContextCleaner: Cleaned accumulator 734
16/09/07 19:11:19 INFO ContextCleaner: Cleaned accumulator 733
16/09/07 19:11:19 INFO BlockManagerInfo: Removed broadcast_1106_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:19 INFO ContextCleaner: Cleaned shuffle 361
16/09/07 19:11:19 INFO MemoryStore: ensureFreeSpace(18849160) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:19 INFO MemoryStore: Block broadcast_1109 stored as values in memory (estimated size 18.0 MB, free 3.7 GB)
16/09/07 19:11:19 INFO MemoryStore: ensureFreeSpace(168000) called with curMem=19392984, maxMem=4042889625
16/09/07 19:11:19 INFO MemoryStore: Block broadcast_1109_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:19 INFO BlockManagerInfo: Added broadcast_1109_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:19 INFO SparkContext: Created broadcast 1109 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:19 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:19 INFO DAGScheduler: Registering RDD 2218 (repartition at SparkUtils.java:190)
16/09/07 19:11:19 INFO DAGScheduler: Got job 372 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:19 INFO DAGScheduler: Final stage: ResultStage 735(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:19 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 734)
16/09/07 19:11:19 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 734)
16/09/07 19:11:19 INFO DAGScheduler: Submitting ShuffleMapStage 734 (MapPartitionsRDD[2218] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:19 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19560984, maxMem=4042889625
16/09/07 19:11:19 INFO MemoryStore: Block broadcast_1110 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:19 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19565448, maxMem=4042889625
16/09/07 19:11:19 INFO MemoryStore: Block broadcast_1110_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:19 INFO BlockManagerInfo: Added broadcast_1110_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:19 INFO SparkContext: Created broadcast 1110 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:19 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 734 (MapPartitionsRDD[2218] at repartition at SparkUtils.java:190)
16/09/07 19:11:19 INFO TaskSchedulerImpl: Adding task set 734.0 with 8 tasks
16/09/07 19:11:19 INFO TaskSetManager: Starting task 0.0 in stage 734.0 (TID 5872, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 1.0 in stage 734.0 (TID 5873, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 2.0 in stage 734.0 (TID 5874, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 3.0 in stage 734.0 (TID 5875, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 4.0 in stage 734.0 (TID 5876, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 5.0 in stage 734.0 (TID 5877, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 6.0 in stage 734.0 (TID 5878, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 7.0 in stage 734.0 (TID 5879, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:19 INFO Executor: Running task 1.0 in stage 734.0 (TID 5873)
16/09/07 19:11:19 INFO Executor: Running task 2.0 in stage 734.0 (TID 5874)
16/09/07 19:11:19 INFO Executor: Running task 5.0 in stage 734.0 (TID 5877)
16/09/07 19:11:19 INFO Executor: Running task 4.0 in stage 734.0 (TID 5876)
16/09/07 19:11:19 INFO Executor: Running task 3.0 in stage 734.0 (TID 5875)
16/09/07 19:11:19 INFO Executor: Running task 0.0 in stage 734.0 (TID 5872)
16/09/07 19:11:19 INFO Executor: Running task 7.0 in stage 734.0 (TID 5879)
16/09/07 19:11:19 INFO Executor: Running task 6.0 in stage 734.0 (TID 5878)
16/09/07 19:11:19 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:19 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:19 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:19 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:19 INFO Executor: Finished task 0.0 in stage 734.0 (TID 5872). 2260 bytes result sent to driver
16/09/07 19:11:19 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:19 INFO TaskSetManager: Finished task 0.0 in stage 734.0 (TID 5872) in 2 ms on localhost (1/8)
16/09/07 19:11:19 INFO Executor: Finished task 7.0 in stage 734.0 (TID 5879). 2260 bytes result sent to driver
16/09/07 19:11:19 INFO Executor: Finished task 4.0 in stage 734.0 (TID 5876). 2260 bytes result sent to driver
16/09/07 19:11:19 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:19 INFO TaskSetManager: Finished task 7.0 in stage 734.0 (TID 5879) in 3 ms on localhost (2/8)
16/09/07 19:11:19 INFO TaskSetManager: Finished task 4.0 in stage 734.0 (TID 5876) in 3 ms on localhost (3/8)
16/09/07 19:11:19 INFO Executor: Finished task 5.0 in stage 734.0 (TID 5877). 2260 bytes result sent to driver
16/09/07 19:11:19 INFO Executor: Finished task 6.0 in stage 734.0 (TID 5878). 2260 bytes result sent to driver
16/09/07 19:11:19 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:19 INFO TaskSetManager: Finished task 5.0 in stage 734.0 (TID 5877) in 3 ms on localhost (4/8)
16/09/07 19:11:19 INFO TaskSetManager: Finished task 6.0 in stage 734.0 (TID 5878) in 3 ms on localhost (5/8)
16/09/07 19:11:19 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:19 INFO Executor: Finished task 1.0 in stage 734.0 (TID 5873). 2260 bytes result sent to driver
16/09/07 19:11:19 INFO TaskSetManager: Finished task 1.0 in stage 734.0 (TID 5873) in 4 ms on localhost (6/8)
16/09/07 19:11:19 INFO Executor: Finished task 2.0 in stage 734.0 (TID 5874). 2260 bytes result sent to driver
16/09/07 19:11:19 INFO Executor: Finished task 3.0 in stage 734.0 (TID 5875). 2260 bytes result sent to driver
16/09/07 19:11:19 INFO TaskSetManager: Finished task 2.0 in stage 734.0 (TID 5874) in 4 ms on localhost (7/8)
16/09/07 19:11:19 INFO TaskSetManager: Finished task 3.0 in stage 734.0 (TID 5875) in 4 ms on localhost (8/8)
16/09/07 19:11:19 INFO TaskSchedulerImpl: Removed TaskSet 734.0, whose tasks have all completed, from pool 
16/09/07 19:11:19 INFO DAGScheduler: ShuffleMapStage 734 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:19 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:19 INFO DAGScheduler: running: Set()
16/09/07 19:11:19 INFO DAGScheduler: waiting: Set(ResultStage 735)
16/09/07 19:11:19 INFO DAGScheduler: failed: Set()
16/09/07 19:11:19 INFO DAGScheduler: Missing parents for ResultStage 735: List()
16/09/07 19:11:19 INFO DAGScheduler: Submitting ResultStage 735 (MapPartitionsRDD[2222] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:19 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19568012, maxMem=4042889625
16/09/07 19:11:19 INFO MemoryStore: Block broadcast_1111 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:19 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19573108, maxMem=4042889625
16/09/07 19:11:19 INFO MemoryStore: Block broadcast_1111_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:19 INFO BlockManagerInfo: Added broadcast_1111_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:19 INFO SparkContext: Created broadcast 1111 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:19 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 735 (MapPartitionsRDD[2222] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:19 INFO TaskSchedulerImpl: Adding task set 735.0 with 8 tasks
16/09/07 19:11:19 INFO TaskSetManager: Starting task 0.0 in stage 735.0 (TID 5880, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 1.0 in stage 735.0 (TID 5881, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 2.0 in stage 735.0 (TID 5882, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 3.0 in stage 735.0 (TID 5883, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 4.0 in stage 735.0 (TID 5884, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 5.0 in stage 735.0 (TID 5885, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 6.0 in stage 735.0 (TID 5886, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:19 INFO TaskSetManager: Starting task 7.0 in stage 735.0 (TID 5887, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:19 INFO Executor: Running task 0.0 in stage 735.0 (TID 5880)
16/09/07 19:11:19 INFO Executor: Running task 4.0 in stage 735.0 (TID 5884)
16/09/07 19:11:19 INFO Executor: Running task 6.0 in stage 735.0 (TID 5886)
16/09/07 19:11:19 INFO Executor: Running task 3.0 in stage 735.0 (TID 5883)
16/09/07 19:11:19 INFO Executor: Running task 5.0 in stage 735.0 (TID 5885)
16/09/07 19:11:19 INFO Executor: Running task 1.0 in stage 735.0 (TID 5881)
16/09/07 19:11:19 INFO Executor: Running task 2.0 in stage 735.0 (TID 5882)
16/09/07 19:11:19 INFO Executor: Running task 7.0 in stage 735.0 (TID 5887)
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:19 INFO Executor: Finished task 4.0 in stage 735.0 (TID 5884). 2160 bytes result sent to driver
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:19 INFO TaskSetManager: Finished task 4.0 in stage 735.0 (TID 5884) in 1 ms on localhost (1/8)
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:19 INFO Executor: Finished task 1.0 in stage 735.0 (TID 5881). 2160 bytes result sent to driver
16/09/07 19:11:19 INFO Executor: Finished task 2.0 in stage 735.0 (TID 5882). 2160 bytes result sent to driver
16/09/07 19:11:19 INFO TaskSetManager: Finished task 1.0 in stage 735.0 (TID 5881) in 1 ms on localhost (2/8)
16/09/07 19:11:19 INFO Executor: Finished task 3.0 in stage 735.0 (TID 5883). 2160 bytes result sent to driver
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:19 INFO TaskSetManager: Finished task 2.0 in stage 735.0 (TID 5882) in 2 ms on localhost (3/8)
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:19 INFO TaskSetManager: Finished task 3.0 in stage 735.0 (TID 5883) in 2 ms on localhost (4/8)
16/09/07 19:11:19 INFO Executor: Finished task 5.0 in stage 735.0 (TID 5885). 2160 bytes result sent to driver
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:19 INFO TaskSetManager: Finished task 5.0 in stage 735.0 (TID 5885) in 2 ms on localhost (5/8)
16/09/07 19:11:19 INFO BlockManagerInfo: Removed broadcast_1110_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:20 INFO Executor: Finished task 6.0 in stage 735.0 (TID 5886). 3485274 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 6.0 in stage 735.0 (TID 5886) in 778 ms on localhost (6/8)
16/09/07 19:11:20 INFO Executor: Finished task 0.0 in stage 735.0 (TID 5880). 3485274 bytes result sent to driver
16/09/07 19:11:20 INFO Executor: Finished task 7.0 in stage 735.0 (TID 5887). 3485274 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 0.0 in stage 735.0 (TID 5880) in 831 ms on localhost (7/8)
16/09/07 19:11:20 INFO TaskSetManager: Finished task 7.0 in stage 735.0 (TID 5887) in 831 ms on localhost (8/8)
16/09/07 19:11:20 INFO DAGScheduler: ResultStage 735 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.832 s
16/09/07 19:11:20 INFO TaskSchedulerImpl: Removed TaskSet 735.0, whose tasks have all completed, from pool 
16/09/07 19:11:20 INFO DAGScheduler: Job 372 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.838938 s
16/09/07 19:11:20 INFO ParameterAveragingTrainingMaster: Completed training of split 51 of 78
16/09/07 19:11:20 INFO ParameterAveragingTrainingMaster: Starting training of split 52 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:20 INFO MemoryStore: ensureFreeSpace(18905576) called with curMem=19568823, maxMem=4042889625
16/09/07 19:11:20 INFO MemoryStore: Block broadcast_1112 stored as values in memory (estimated size 18.0 MB, free 3.7 GB)
16/09/07 19:11:20 INFO MemoryStore: ensureFreeSpace(167963) called with curMem=38474399, maxMem=4042889625
16/09/07 19:11:20 INFO MemoryStore: Block broadcast_1112_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:20 INFO BlockManagerInfo: Added broadcast_1112_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:20 INFO SparkContext: Created broadcast 1112 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:20 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:20 INFO DAGScheduler: Registering RDD 2223 (repartition at SparkUtils.java:190)
16/09/07 19:11:20 INFO DAGScheduler: Got job 373 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:20 INFO DAGScheduler: Final stage: ResultStage 737(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:20 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 736)
16/09/07 19:11:20 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 736)
16/09/07 19:11:20 INFO DAGScheduler: Submitting ShuffleMapStage 736 (MapPartitionsRDD[2223] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:20 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=38642362, maxMem=4042889625
16/09/07 19:11:20 INFO MemoryStore: Block broadcast_1113 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:20 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=38646826, maxMem=4042889625
16/09/07 19:11:20 INFO MemoryStore: Block broadcast_1113_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:20 INFO BlockManagerInfo: Added broadcast_1113_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:20 INFO SparkContext: Created broadcast 1113 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:20 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 736 (MapPartitionsRDD[2223] at repartition at SparkUtils.java:190)
16/09/07 19:11:20 INFO TaskSchedulerImpl: Adding task set 736.0 with 8 tasks
16/09/07 19:11:20 INFO TaskSetManager: Starting task 0.0 in stage 736.0 (TID 5888, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 1.0 in stage 736.0 (TID 5889, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 2.0 in stage 736.0 (TID 5890, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 3.0 in stage 736.0 (TID 5891, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 4.0 in stage 736.0 (TID 5892, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 5.0 in stage 736.0 (TID 5893, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 6.0 in stage 736.0 (TID 5894, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 7.0 in stage 736.0 (TID 5895, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:20 INFO Executor: Running task 0.0 in stage 736.0 (TID 5888)
16/09/07 19:11:20 INFO Executor: Running task 3.0 in stage 736.0 (TID 5891)
16/09/07 19:11:20 INFO Executor: Running task 7.0 in stage 736.0 (TID 5895)
16/09/07 19:11:20 INFO Executor: Running task 5.0 in stage 736.0 (TID 5893)
16/09/07 19:11:20 INFO Executor: Running task 6.0 in stage 736.0 (TID 5894)
16/09/07 19:11:20 INFO Executor: Running task 2.0 in stage 736.0 (TID 5890)
16/09/07 19:11:20 INFO Executor: Running task 1.0 in stage 736.0 (TID 5889)
16/09/07 19:11:20 INFO Executor: Running task 4.0 in stage 736.0 (TID 5892)
16/09/07 19:11:20 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:20 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:20 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:20 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:20 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:20 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:20 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:20 INFO Executor: Finished task 2.0 in stage 736.0 (TID 5890). 2260 bytes result sent to driver
16/09/07 19:11:20 INFO Executor: Finished task 5.0 in stage 736.0 (TID 5893). 2260 bytes result sent to driver
16/09/07 19:11:20 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:20 INFO Executor: Finished task 6.0 in stage 736.0 (TID 5894). 2260 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 2.0 in stage 736.0 (TID 5890) in 2 ms on localhost (1/8)
16/09/07 19:11:20 INFO TaskSetManager: Finished task 5.0 in stage 736.0 (TID 5893) in 2 ms on localhost (2/8)
16/09/07 19:11:20 INFO Executor: Finished task 3.0 in stage 736.0 (TID 5891). 2260 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 6.0 in stage 736.0 (TID 5894) in 2 ms on localhost (3/8)
16/09/07 19:11:20 INFO Executor: Finished task 0.0 in stage 736.0 (TID 5888). 2260 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 3.0 in stage 736.0 (TID 5891) in 2 ms on localhost (4/8)
16/09/07 19:11:20 INFO TaskSetManager: Finished task 0.0 in stage 736.0 (TID 5888) in 3 ms on localhost (5/8)
16/09/07 19:11:20 INFO Executor: Finished task 4.0 in stage 736.0 (TID 5892). 2260 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 4.0 in stage 736.0 (TID 5892) in 3 ms on localhost (6/8)
16/09/07 19:11:20 INFO Executor: Finished task 1.0 in stage 736.0 (TID 5889). 2260 bytes result sent to driver
16/09/07 19:11:20 INFO Executor: Finished task 7.0 in stage 736.0 (TID 5895). 2260 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 1.0 in stage 736.0 (TID 5889) in 3 ms on localhost (7/8)
16/09/07 19:11:20 INFO TaskSetManager: Finished task 7.0 in stage 736.0 (TID 5895) in 3 ms on localhost (8/8)
16/09/07 19:11:20 INFO TaskSchedulerImpl: Removed TaskSet 736.0, whose tasks have all completed, from pool 
16/09/07 19:11:20 INFO DAGScheduler: ShuffleMapStage 736 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:20 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:20 INFO DAGScheduler: running: Set()
16/09/07 19:11:20 INFO DAGScheduler: waiting: Set(ResultStage 737)
16/09/07 19:11:20 INFO DAGScheduler: failed: Set()
16/09/07 19:11:20 INFO DAGScheduler: Missing parents for ResultStage 737: List()
16/09/07 19:11:20 INFO DAGScheduler: Submitting ResultStage 737 (MapPartitionsRDD[2227] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:20 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=38649390, maxMem=4042889625
16/09/07 19:11:20 INFO MemoryStore: Block broadcast_1114 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:20 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=38654486, maxMem=4042889625
16/09/07 19:11:20 INFO MemoryStore: Block broadcast_1114_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:20 INFO BlockManagerInfo: Added broadcast_1114_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:20 INFO SparkContext: Created broadcast 1114 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:20 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 737 (MapPartitionsRDD[2227] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:20 INFO TaskSchedulerImpl: Adding task set 737.0 with 8 tasks
16/09/07 19:11:20 INFO TaskSetManager: Starting task 0.0 in stage 737.0 (TID 5896, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 1.0 in stage 737.0 (TID 5897, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 2.0 in stage 737.0 (TID 5898, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 3.0 in stage 737.0 (TID 5899, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 4.0 in stage 737.0 (TID 5900, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 5.0 in stage 737.0 (TID 5901, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 6.0 in stage 737.0 (TID 5902, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:20 INFO TaskSetManager: Starting task 7.0 in stage 737.0 (TID 5903, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:20 INFO Executor: Running task 0.0 in stage 737.0 (TID 5896)
16/09/07 19:11:20 INFO Executor: Running task 5.0 in stage 737.0 (TID 5901)
16/09/07 19:11:20 INFO Executor: Running task 6.0 in stage 737.0 (TID 5902)
16/09/07 19:11:20 INFO Executor: Running task 1.0 in stage 737.0 (TID 5897)
16/09/07 19:11:20 INFO Executor: Running task 2.0 in stage 737.0 (TID 5898)
16/09/07 19:11:20 INFO Executor: Running task 7.0 in stage 737.0 (TID 5903)
16/09/07 19:11:20 INFO Executor: Running task 4.0 in stage 737.0 (TID 5900)
16/09/07 19:11:20 INFO Executor: Running task 3.0 in stage 737.0 (TID 5899)
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:20 INFO Executor: Finished task 3.0 in stage 737.0 (TID 5899). 2160 bytes result sent to driver
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:20 INFO TaskSetManager: Finished task 3.0 in stage 737.0 (TID 5899) in 2 ms on localhost (1/8)
16/09/07 19:11:20 INFO Executor: Finished task 4.0 in stage 737.0 (TID 5900). 2160 bytes result sent to driver
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:20 INFO TaskSetManager: Finished task 4.0 in stage 737.0 (TID 5900) in 2 ms on localhost (2/8)
16/09/07 19:11:20 INFO Executor: Finished task 1.0 in stage 737.0 (TID 5897). 2160 bytes result sent to driver
16/09/07 19:11:20 INFO Executor: Finished task 5.0 in stage 737.0 (TID 5901). 2160 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 1.0 in stage 737.0 (TID 5897) in 3 ms on localhost (3/8)
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:20 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:20 INFO TaskSetManager: Finished task 5.0 in stage 737.0 (TID 5901) in 3 ms on localhost (4/8)
16/09/07 19:11:20 INFO Executor: Finished task 2.0 in stage 737.0 (TID 5898). 2160 bytes result sent to driver
16/09/07 19:11:20 INFO TaskSetManager: Finished task 2.0 in stage 737.0 (TID 5898) in 3 ms on localhost (5/8)
16/09/07 19:11:20 INFO BlockManagerInfo: Removed broadcast_1113_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:20 INFO BlockManagerInfo: Removed broadcast_1111_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:20 INFO ContextCleaner: Cleaned accumulator 736
16/09/07 19:11:20 INFO ContextCleaner: Cleaned accumulator 735
16/09/07 19:11:20 INFO BlockManagerInfo: Removed broadcast_1109_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:20 INFO ContextCleaner: Cleaned shuffle 362
16/09/07 19:11:21 INFO Executor: Finished task 6.0 in stage 737.0 (TID 5902). 3485274 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 6.0 in stage 737.0 (TID 5902) in 900 ms on localhost (6/8)
16/09/07 19:11:21 INFO Executor: Finished task 0.0 in stage 737.0 (TID 5896). 3485274 bytes result sent to driver
16/09/07 19:11:21 INFO Executor: Finished task 7.0 in stage 737.0 (TID 5903). 3485274 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 0.0 in stage 737.0 (TID 5896) in 956 ms on localhost (7/8)
16/09/07 19:11:21 INFO TaskSetManager: Finished task 7.0 in stage 737.0 (TID 5903) in 955 ms on localhost (8/8)
16/09/07 19:11:21 INFO TaskSchedulerImpl: Removed TaskSet 737.0, whose tasks have all completed, from pool 
16/09/07 19:11:21 INFO DAGScheduler: ResultStage 737 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.956 s
16/09/07 19:11:21 INFO DAGScheduler: Job 373 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.962634 s
16/09/07 19:11:21 INFO ParameterAveragingTrainingMaster: Completed training of split 52 of 78
16/09/07 19:11:21 INFO ParameterAveragingTrainingMaster: Starting training of split 53 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:21 INFO MemoryStore: ensureFreeSpace(18931456) called with curMem=19625202, maxMem=4042889625
16/09/07 19:11:21 INFO MemoryStore: Block broadcast_1115 stored as values in memory (estimated size 18.1 MB, free 3.7 GB)
16/09/07 19:11:21 INFO BlockManagerInfo: Removed broadcast_1114_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:21 INFO ContextCleaner: Cleaned accumulator 738
16/09/07 19:11:21 INFO ContextCleaner: Cleaned accumulator 737
16/09/07 19:11:21 INFO BlockManagerInfo: Removed broadcast_1112_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:21 INFO ContextCleaner: Cleaned shuffle 363
16/09/07 19:11:21 INFO MemoryStore: ensureFreeSpace(167946) called with curMem=19475280, maxMem=4042889625
16/09/07 19:11:21 INFO MemoryStore: Block broadcast_1115_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:21 INFO BlockManagerInfo: Added broadcast_1115_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:21 INFO SparkContext: Created broadcast 1115 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:21 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:21 INFO DAGScheduler: Registering RDD 2228 (repartition at SparkUtils.java:190)
16/09/07 19:11:21 INFO DAGScheduler: Got job 374 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:21 INFO DAGScheduler: Final stage: ResultStage 739(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:21 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 738)
16/09/07 19:11:21 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 738)
16/09/07 19:11:21 INFO DAGScheduler: Submitting ShuffleMapStage 738 (MapPartitionsRDD[2228] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:21 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19643226, maxMem=4042889625
16/09/07 19:11:21 INFO MemoryStore: Block broadcast_1116 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:21 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19647690, maxMem=4042889625
16/09/07 19:11:21 INFO MemoryStore: Block broadcast_1116_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:21 INFO BlockManagerInfo: Added broadcast_1116_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:21 INFO SparkContext: Created broadcast 1116 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:21 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 738 (MapPartitionsRDD[2228] at repartition at SparkUtils.java:190)
16/09/07 19:11:21 INFO TaskSchedulerImpl: Adding task set 738.0 with 8 tasks
16/09/07 19:11:21 INFO TaskSetManager: Starting task 0.0 in stage 738.0 (TID 5904, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 1.0 in stage 738.0 (TID 5905, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 2.0 in stage 738.0 (TID 5906, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 3.0 in stage 738.0 (TID 5907, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 4.0 in stage 738.0 (TID 5908, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 5.0 in stage 738.0 (TID 5909, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 6.0 in stage 738.0 (TID 5910, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 7.0 in stage 738.0 (TID 5911, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:21 INFO Executor: Running task 2.0 in stage 738.0 (TID 5906)
16/09/07 19:11:21 INFO Executor: Running task 0.0 in stage 738.0 (TID 5904)
16/09/07 19:11:21 INFO Executor: Running task 6.0 in stage 738.0 (TID 5910)
16/09/07 19:11:21 INFO Executor: Running task 1.0 in stage 738.0 (TID 5905)
16/09/07 19:11:21 INFO Executor: Running task 3.0 in stage 738.0 (TID 5907)
16/09/07 19:11:21 INFO Executor: Running task 5.0 in stage 738.0 (TID 5909)
16/09/07 19:11:21 INFO Executor: Running task 4.0 in stage 738.0 (TID 5908)
16/09/07 19:11:21 INFO Executor: Running task 7.0 in stage 738.0 (TID 5911)
16/09/07 19:11:21 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:21 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:21 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:21 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:21 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:21 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:21 INFO Executor: Finished task 5.0 in stage 738.0 (TID 5909). 2260 bytes result sent to driver
16/09/07 19:11:21 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:21 INFO Executor: Finished task 6.0 in stage 738.0 (TID 5910). 2260 bytes result sent to driver
16/09/07 19:11:21 INFO Executor: Finished task 1.0 in stage 738.0 (TID 5905). 2260 bytes result sent to driver
16/09/07 19:11:21 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:21 INFO TaskSetManager: Finished task 5.0 in stage 738.0 (TID 5909) in 2 ms on localhost (1/8)
16/09/07 19:11:21 INFO Executor: Finished task 3.0 in stage 738.0 (TID 5907). 2260 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 6.0 in stage 738.0 (TID 5910) in 2 ms on localhost (2/8)
16/09/07 19:11:21 INFO TaskSetManager: Finished task 1.0 in stage 738.0 (TID 5905) in 3 ms on localhost (3/8)
16/09/07 19:11:21 INFO Executor: Finished task 0.0 in stage 738.0 (TID 5904). 2260 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 3.0 in stage 738.0 (TID 5907) in 3 ms on localhost (4/8)
16/09/07 19:11:21 INFO TaskSetManager: Finished task 0.0 in stage 738.0 (TID 5904) in 3 ms on localhost (5/8)
16/09/07 19:11:21 INFO Executor: Finished task 7.0 in stage 738.0 (TID 5911). 2260 bytes result sent to driver
16/09/07 19:11:21 INFO Executor: Finished task 2.0 in stage 738.0 (TID 5906). 2260 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 7.0 in stage 738.0 (TID 5911) in 2 ms on localhost (6/8)
16/09/07 19:11:21 INFO TaskSetManager: Finished task 2.0 in stage 738.0 (TID 5906) in 3 ms on localhost (7/8)
16/09/07 19:11:21 INFO Executor: Finished task 4.0 in stage 738.0 (TID 5908). 2260 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 4.0 in stage 738.0 (TID 5908) in 3 ms on localhost (8/8)
16/09/07 19:11:21 INFO TaskSchedulerImpl: Removed TaskSet 738.0, whose tasks have all completed, from pool 
16/09/07 19:11:21 INFO DAGScheduler: ShuffleMapStage 738 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:21 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:21 INFO DAGScheduler: running: Set()
16/09/07 19:11:21 INFO DAGScheduler: waiting: Set(ResultStage 739)
16/09/07 19:11:21 INFO DAGScheduler: failed: Set()
16/09/07 19:11:21 INFO DAGScheduler: Missing parents for ResultStage 739: List()
16/09/07 19:11:21 INFO DAGScheduler: Submitting ResultStage 739 (MapPartitionsRDD[2232] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:21 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19650254, maxMem=4042889625
16/09/07 19:11:21 INFO MemoryStore: Block broadcast_1117 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:21 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19655350, maxMem=4042889625
16/09/07 19:11:21 INFO MemoryStore: Block broadcast_1117_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:21 INFO BlockManagerInfo: Added broadcast_1117_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:21 INFO SparkContext: Created broadcast 1117 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:21 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 739 (MapPartitionsRDD[2232] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:21 INFO TaskSchedulerImpl: Adding task set 739.0 with 8 tasks
16/09/07 19:11:21 INFO TaskSetManager: Starting task 0.0 in stage 739.0 (TID 5912, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 1.0 in stage 739.0 (TID 5913, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 2.0 in stage 739.0 (TID 5914, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 3.0 in stage 739.0 (TID 5915, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 4.0 in stage 739.0 (TID 5916, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 5.0 in stage 739.0 (TID 5917, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 6.0 in stage 739.0 (TID 5918, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:21 INFO TaskSetManager: Starting task 7.0 in stage 739.0 (TID 5919, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:21 INFO Executor: Running task 0.0 in stage 739.0 (TID 5912)
16/09/07 19:11:21 INFO Executor: Running task 4.0 in stage 739.0 (TID 5916)
16/09/07 19:11:21 INFO Executor: Running task 3.0 in stage 739.0 (TID 5915)
16/09/07 19:11:21 INFO Executor: Running task 2.0 in stage 739.0 (TID 5914)
16/09/07 19:11:21 INFO Executor: Running task 7.0 in stage 739.0 (TID 5919)
16/09/07 19:11:21 INFO Executor: Running task 6.0 in stage 739.0 (TID 5918)
16/09/07 19:11:21 INFO Executor: Running task 1.0 in stage 739.0 (TID 5913)
16/09/07 19:11:21 INFO Executor: Running task 5.0 in stage 739.0 (TID 5917)
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:21 INFO Executor: Finished task 2.0 in stage 739.0 (TID 5914). 2160 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 2.0 in stage 739.0 (TID 5914) in 2 ms on localhost (1/8)
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:21 INFO Executor: Finished task 1.0 in stage 739.0 (TID 5913). 2160 bytes result sent to driver
16/09/07 19:11:21 INFO Executor: Finished task 5.0 in stage 739.0 (TID 5917). 2160 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 1.0 in stage 739.0 (TID 5913) in 2 ms on localhost (2/8)
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:21 INFO TaskSetManager: Finished task 5.0 in stage 739.0 (TID 5917) in 2 ms on localhost (3/8)
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:21 INFO Executor: Finished task 3.0 in stage 739.0 (TID 5915). 2160 bytes result sent to driver
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:21 INFO TaskSetManager: Finished task 3.0 in stage 739.0 (TID 5915) in 2 ms on localhost (4/8)
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:21 INFO Executor: Finished task 4.0 in stage 739.0 (TID 5916). 2160 bytes result sent to driver
16/09/07 19:11:21 INFO TaskSetManager: Finished task 4.0 in stage 739.0 (TID 5916) in 3 ms on localhost (5/8)
16/09/07 19:11:21 INFO BlockManagerInfo: Removed broadcast_1116_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:22 INFO Executor: Finished task 6.0 in stage 739.0 (TID 5918). 3485524 bytes result sent to driver
16/09/07 19:11:22 INFO TaskSetManager: Finished task 6.0 in stage 739.0 (TID 5918) in 892 ms on localhost (6/8)
16/09/07 19:11:22 INFO Executor: Finished task 7.0 in stage 739.0 (TID 5919). 3485524 bytes result sent to driver
16/09/07 19:11:22 INFO Executor: Finished task 0.0 in stage 739.0 (TID 5912). 3485524 bytes result sent to driver
16/09/07 19:11:22 INFO TaskSetManager: Finished task 0.0 in stage 739.0 (TID 5912) in 1064 ms on localhost (7/8)
16/09/07 19:11:22 INFO TaskSetManager: Finished task 7.0 in stage 739.0 (TID 5919) in 1065 ms on localhost (8/8)
16/09/07 19:11:22 INFO TaskSchedulerImpl: Removed TaskSet 739.0, whose tasks have all completed, from pool 
16/09/07 19:11:22 INFO DAGScheduler: ResultStage 739 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 1.065 s
16/09/07 19:11:22 INFO DAGScheduler: Job 374 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.072456 s
16/09/07 19:11:22 INFO ParameterAveragingTrainingMaster: Completed training of split 53 of 78
16/09/07 19:11:22 INFO ParameterAveragingTrainingMaster: Starting training of split 54 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:23 INFO MemoryStore: ensureFreeSpace(18977232) called with curMem=19651065, maxMem=4042889625
16/09/07 19:11:23 INFO MemoryStore: Block broadcast_1118 stored as values in memory (estimated size 18.1 MB, free 3.7 GB)
16/09/07 19:11:23 INFO MemoryStore: ensureFreeSpace(167971) called with curMem=38628297, maxMem=4042889625
16/09/07 19:11:23 INFO MemoryStore: Block broadcast_1118_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:23 INFO BlockManagerInfo: Added broadcast_1118_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:23 INFO SparkContext: Created broadcast 1118 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:23 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:23 INFO DAGScheduler: Registering RDD 2233 (repartition at SparkUtils.java:190)
16/09/07 19:11:23 INFO DAGScheduler: Got job 375 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:23 INFO DAGScheduler: Final stage: ResultStage 741(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:23 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 740)
16/09/07 19:11:23 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 740)
16/09/07 19:11:23 INFO DAGScheduler: Submitting ShuffleMapStage 740 (MapPartitionsRDD[2233] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:23 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=38796268, maxMem=4042889625
16/09/07 19:11:23 INFO MemoryStore: Block broadcast_1119 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:23 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=38800732, maxMem=4042889625
16/09/07 19:11:23 INFO MemoryStore: Block broadcast_1119_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:23 INFO BlockManagerInfo: Added broadcast_1119_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:23 INFO SparkContext: Created broadcast 1119 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:23 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 740 (MapPartitionsRDD[2233] at repartition at SparkUtils.java:190)
16/09/07 19:11:23 INFO TaskSchedulerImpl: Adding task set 740.0 with 8 tasks
16/09/07 19:11:23 INFO TaskSetManager: Starting task 0.0 in stage 740.0 (TID 5920, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 1.0 in stage 740.0 (TID 5921, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 2.0 in stage 740.0 (TID 5922, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 3.0 in stage 740.0 (TID 5923, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 4.0 in stage 740.0 (TID 5924, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 5.0 in stage 740.0 (TID 5925, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 6.0 in stage 740.0 (TID 5926, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 7.0 in stage 740.0 (TID 5927, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:23 INFO Executor: Running task 1.0 in stage 740.0 (TID 5921)
16/09/07 19:11:23 INFO Executor: Running task 0.0 in stage 740.0 (TID 5920)
16/09/07 19:11:23 INFO Executor: Running task 2.0 in stage 740.0 (TID 5922)
16/09/07 19:11:23 INFO Executor: Running task 5.0 in stage 740.0 (TID 5925)
16/09/07 19:11:23 INFO Executor: Running task 4.0 in stage 740.0 (TID 5924)
16/09/07 19:11:23 INFO Executor: Running task 7.0 in stage 740.0 (TID 5927)
16/09/07 19:11:23 INFO Executor: Running task 6.0 in stage 740.0 (TID 5926)
16/09/07 19:11:23 INFO Executor: Running task 3.0 in stage 740.0 (TID 5923)
16/09/07 19:11:23 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:23 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:23 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:23 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:23 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:23 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:23 INFO Executor: Finished task 2.0 in stage 740.0 (TID 5922). 2260 bytes result sent to driver
16/09/07 19:11:23 INFO Executor: Finished task 0.0 in stage 740.0 (TID 5920). 2260 bytes result sent to driver
16/09/07 19:11:23 INFO Executor: Finished task 3.0 in stage 740.0 (TID 5923). 2260 bytes result sent to driver
16/09/07 19:11:23 INFO Executor: Finished task 4.0 in stage 740.0 (TID 5924). 2260 bytes result sent to driver
16/09/07 19:11:23 INFO Executor: Finished task 5.0 in stage 740.0 (TID 5925). 2260 bytes result sent to driver
16/09/07 19:11:23 INFO TaskSetManager: Finished task 2.0 in stage 740.0 (TID 5922) in 3 ms on localhost (1/8)
16/09/07 19:11:23 INFO TaskSetManager: Finished task 0.0 in stage 740.0 (TID 5920) in 3 ms on localhost (2/8)
16/09/07 19:11:23 INFO Executor: Finished task 6.0 in stage 740.0 (TID 5926). 2260 bytes result sent to driver
16/09/07 19:11:23 INFO TaskSetManager: Finished task 3.0 in stage 740.0 (TID 5923) in 3 ms on localhost (3/8)
16/09/07 19:11:23 INFO TaskSetManager: Finished task 4.0 in stage 740.0 (TID 5924) in 3 ms on localhost (4/8)
16/09/07 19:11:23 INFO TaskSetManager: Finished task 6.0 in stage 740.0 (TID 5926) in 3 ms on localhost (5/8)
16/09/07 19:11:23 INFO TaskSetManager: Finished task 5.0 in stage 740.0 (TID 5925) in 3 ms on localhost (6/8)
16/09/07 19:11:23 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:23 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:23 INFO Executor: Finished task 1.0 in stage 740.0 (TID 5921). 2260 bytes result sent to driver
16/09/07 19:11:23 INFO Executor: Finished task 7.0 in stage 740.0 (TID 5927). 2260 bytes result sent to driver
16/09/07 19:11:23 INFO TaskSetManager: Finished task 1.0 in stage 740.0 (TID 5921) in 4 ms on localhost (7/8)
16/09/07 19:11:23 INFO TaskSetManager: Finished task 7.0 in stage 740.0 (TID 5927) in 4 ms on localhost (8/8)
16/09/07 19:11:23 INFO TaskSchedulerImpl: Removed TaskSet 740.0, whose tasks have all completed, from pool 
16/09/07 19:11:23 INFO DAGScheduler: ShuffleMapStage 740 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:23 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:23 INFO DAGScheduler: running: Set()
16/09/07 19:11:23 INFO DAGScheduler: waiting: Set(ResultStage 741)
16/09/07 19:11:23 INFO DAGScheduler: failed: Set()
16/09/07 19:11:23 INFO DAGScheduler: Missing parents for ResultStage 741: List()
16/09/07 19:11:23 INFO DAGScheduler: Submitting ResultStage 741 (MapPartitionsRDD[2237] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:23 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=38803296, maxMem=4042889625
16/09/07 19:11:23 INFO MemoryStore: Block broadcast_1120 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:23 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=38808392, maxMem=4042889625
16/09/07 19:11:23 INFO MemoryStore: Block broadcast_1120_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:23 INFO BlockManagerInfo: Added broadcast_1120_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:23 INFO SparkContext: Created broadcast 1120 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:23 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 741 (MapPartitionsRDD[2237] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:23 INFO TaskSchedulerImpl: Adding task set 741.0 with 8 tasks
16/09/07 19:11:23 INFO TaskSetManager: Starting task 0.0 in stage 741.0 (TID 5928, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 1.0 in stage 741.0 (TID 5929, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 2.0 in stage 741.0 (TID 5930, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 3.0 in stage 741.0 (TID 5931, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 4.0 in stage 741.0 (TID 5932, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 5.0 in stage 741.0 (TID 5933, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 6.0 in stage 741.0 (TID 5934, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:23 INFO TaskSetManager: Starting task 7.0 in stage 741.0 (TID 5935, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:23 INFO Executor: Running task 0.0 in stage 741.0 (TID 5928)
16/09/07 19:11:23 INFO Executor: Running task 2.0 in stage 741.0 (TID 5930)
16/09/07 19:11:23 INFO Executor: Running task 7.0 in stage 741.0 (TID 5935)
16/09/07 19:11:23 INFO Executor: Running task 1.0 in stage 741.0 (TID 5929)
16/09/07 19:11:23 INFO Executor: Running task 6.0 in stage 741.0 (TID 5934)
16/09/07 19:11:23 INFO Executor: Running task 4.0 in stage 741.0 (TID 5932)
16/09/07 19:11:23 INFO Executor: Running task 3.0 in stage 741.0 (TID 5931)
16/09/07 19:11:23 INFO Executor: Running task 5.0 in stage 741.0 (TID 5933)
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:23 INFO Executor: Finished task 5.0 in stage 741.0 (TID 5933). 2160 bytes result sent to driver
16/09/07 19:11:23 INFO Executor: Finished task 2.0 in stage 741.0 (TID 5930). 2160 bytes result sent to driver
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:23 INFO TaskSetManager: Finished task 2.0 in stage 741.0 (TID 5930) in 2 ms on localhost (1/8)
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:23 INFO Executor: Finished task 3.0 in stage 741.0 (TID 5931). 2160 bytes result sent to driver
16/09/07 19:11:23 INFO TaskSetManager: Finished task 5.0 in stage 741.0 (TID 5933) in 1 ms on localhost (2/8)
16/09/07 19:11:23 INFO TaskSetManager: Finished task 3.0 in stage 741.0 (TID 5931) in 2 ms on localhost (3/8)
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:23 INFO Executor: Finished task 1.0 in stage 741.0 (TID 5929). 2160 bytes result sent to driver
16/09/07 19:11:23 INFO Executor: Finished task 4.0 in stage 741.0 (TID 5932). 2160 bytes result sent to driver
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:23 INFO TaskSetManager: Finished task 1.0 in stage 741.0 (TID 5929) in 3 ms on localhost (4/8)
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:23 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:23 INFO TaskSetManager: Finished task 4.0 in stage 741.0 (TID 5932) in 2 ms on localhost (5/8)
16/09/07 19:11:23 INFO BlockManagerInfo: Removed broadcast_1119_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:23 INFO BlockManagerInfo: Removed broadcast_1117_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:23 INFO ContextCleaner: Cleaned accumulator 740
16/09/07 19:11:23 INFO ContextCleaner: Cleaned accumulator 739
16/09/07 19:11:23 INFO BlockManagerInfo: Removed broadcast_1115_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:23 INFO ContextCleaner: Cleaned shuffle 364
16/09/07 19:11:23 INFO Executor: Finished task 6.0 in stage 741.0 (TID 5934). 3485274 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 6.0 in stage 741.0 (TID 5934) in 868 ms on localhost (6/8)
16/09/07 19:11:24 INFO Executor: Finished task 7.0 in stage 741.0 (TID 5935). 3485274 bytes result sent to driver
16/09/07 19:11:24 INFO Executor: Finished task 0.0 in stage 741.0 (TID 5928). 3485274 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 7.0 in stage 741.0 (TID 5935) in 918 ms on localhost (7/8)
16/09/07 19:11:24 INFO TaskSetManager: Finished task 0.0 in stage 741.0 (TID 5928) in 922 ms on localhost (8/8)
16/09/07 19:11:24 INFO TaskSchedulerImpl: Removed TaskSet 741.0, whose tasks have all completed, from pool 
16/09/07 19:11:24 INFO DAGScheduler: ResultStage 741 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.922 s
16/09/07 19:11:24 INFO DAGScheduler: Job 375 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.929190 s
16/09/07 19:11:24 INFO ParameterAveragingTrainingMaster: Completed training of split 54 of 78
16/09/07 19:11:24 INFO ParameterAveragingTrainingMaster: Starting training of split 55 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:24 INFO MemoryStore: ensureFreeSpace(18988192) called with curMem=19696866, maxMem=4042889625
16/09/07 19:11:24 INFO MemoryStore: Block broadcast_1121 stored as values in memory (estimated size 18.1 MB, free 3.7 GB)
16/09/07 19:11:24 INFO BlockManagerInfo: Removed broadcast_1120_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:24 INFO ContextCleaner: Cleaned accumulator 742
16/09/07 19:11:24 INFO ContextCleaner: Cleaned accumulator 741
16/09/07 19:11:24 INFO BlockManagerInfo: Removed broadcast_1118_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:24 INFO ContextCleaner: Cleaned shuffle 365
16/09/07 19:11:24 INFO MemoryStore: ensureFreeSpace(167955) called with curMem=19532016, maxMem=4042889625
16/09/07 19:11:24 INFO MemoryStore: Block broadcast_1121_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:24 INFO BlockManagerInfo: Added broadcast_1121_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:24 INFO SparkContext: Created broadcast 1121 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:24 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:24 INFO DAGScheduler: Registering RDD 2238 (repartition at SparkUtils.java:190)
16/09/07 19:11:24 INFO DAGScheduler: Got job 376 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:24 INFO DAGScheduler: Final stage: ResultStage 743(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:24 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 742)
16/09/07 19:11:24 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 742)
16/09/07 19:11:24 INFO DAGScheduler: Submitting ShuffleMapStage 742 (MapPartitionsRDD[2238] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:24 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19699971, maxMem=4042889625
16/09/07 19:11:24 INFO MemoryStore: Block broadcast_1122 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:24 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=19704435, maxMem=4042889625
16/09/07 19:11:24 INFO MemoryStore: Block broadcast_1122_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:24 INFO BlockManagerInfo: Added broadcast_1122_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:24 INFO SparkContext: Created broadcast 1122 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:24 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 742 (MapPartitionsRDD[2238] at repartition at SparkUtils.java:190)
16/09/07 19:11:24 INFO TaskSchedulerImpl: Adding task set 742.0 with 8 tasks
16/09/07 19:11:24 INFO TaskSetManager: Starting task 0.0 in stage 742.0 (TID 5936, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 1.0 in stage 742.0 (TID 5937, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 2.0 in stage 742.0 (TID 5938, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 3.0 in stage 742.0 (TID 5939, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 4.0 in stage 742.0 (TID 5940, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 5.0 in stage 742.0 (TID 5941, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 6.0 in stage 742.0 (TID 5942, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 7.0 in stage 742.0 (TID 5943, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:24 INFO Executor: Running task 1.0 in stage 742.0 (TID 5937)
16/09/07 19:11:24 INFO Executor: Running task 0.0 in stage 742.0 (TID 5936)
16/09/07 19:11:24 INFO Executor: Running task 3.0 in stage 742.0 (TID 5939)
16/09/07 19:11:24 INFO Executor: Running task 7.0 in stage 742.0 (TID 5943)
16/09/07 19:11:24 INFO Executor: Running task 5.0 in stage 742.0 (TID 5941)
16/09/07 19:11:24 INFO Executor: Running task 6.0 in stage 742.0 (TID 5942)
16/09/07 19:11:24 INFO Executor: Running task 2.0 in stage 742.0 (TID 5938)
16/09/07 19:11:24 INFO Executor: Running task 4.0 in stage 742.0 (TID 5940)
16/09/07 19:11:24 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:24 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:24 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:24 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:24 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:24 INFO Executor: Finished task 5.0 in stage 742.0 (TID 5941). 2260 bytes result sent to driver
16/09/07 19:11:24 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:24 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:24 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:24 INFO TaskSetManager: Finished task 5.0 in stage 742.0 (TID 5941) in 3 ms on localhost (1/8)
16/09/07 19:11:24 INFO Executor: Finished task 2.0 in stage 742.0 (TID 5938). 2260 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 2.0 in stage 742.0 (TID 5938) in 3 ms on localhost (2/8)
16/09/07 19:11:24 INFO Executor: Finished task 4.0 in stage 742.0 (TID 5940). 2260 bytes result sent to driver
16/09/07 19:11:24 INFO Executor: Finished task 0.0 in stage 742.0 (TID 5936). 2260 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 4.0 in stage 742.0 (TID 5940) in 4 ms on localhost (3/8)
16/09/07 19:11:24 INFO TaskSetManager: Finished task 0.0 in stage 742.0 (TID 5936) in 4 ms on localhost (4/8)
16/09/07 19:11:24 INFO Executor: Finished task 3.0 in stage 742.0 (TID 5939). 2260 bytes result sent to driver
16/09/07 19:11:24 INFO Executor: Finished task 1.0 in stage 742.0 (TID 5937). 2260 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 3.0 in stage 742.0 (TID 5939) in 4 ms on localhost (5/8)
16/09/07 19:11:24 INFO TaskSetManager: Finished task 1.0 in stage 742.0 (TID 5937) in 4 ms on localhost (6/8)
16/09/07 19:11:24 INFO Executor: Finished task 7.0 in stage 742.0 (TID 5943). 2260 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 7.0 in stage 742.0 (TID 5943) in 4 ms on localhost (7/8)
16/09/07 19:11:24 INFO Executor: Finished task 6.0 in stage 742.0 (TID 5942). 2260 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 6.0 in stage 742.0 (TID 5942) in 5 ms on localhost (8/8)
16/09/07 19:11:24 INFO TaskSchedulerImpl: Removed TaskSet 742.0, whose tasks have all completed, from pool 
16/09/07 19:11:24 INFO DAGScheduler: ShuffleMapStage 742 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:11:24 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:24 INFO DAGScheduler: running: Set()
16/09/07 19:11:24 INFO DAGScheduler: waiting: Set(ResultStage 743)
16/09/07 19:11:24 INFO DAGScheduler: failed: Set()
16/09/07 19:11:24 INFO DAGScheduler: Missing parents for ResultStage 743: List()
16/09/07 19:11:24 INFO DAGScheduler: Submitting ResultStage 743 (MapPartitionsRDD[2242] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:24 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19706998, maxMem=4042889625
16/09/07 19:11:24 INFO MemoryStore: Block broadcast_1123 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:24 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19712094, maxMem=4042889625
16/09/07 19:11:24 INFO MemoryStore: Block broadcast_1123_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:24 INFO BlockManagerInfo: Added broadcast_1123_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:24 INFO SparkContext: Created broadcast 1123 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:24 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 743 (MapPartitionsRDD[2242] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:24 INFO TaskSchedulerImpl: Adding task set 743.0 with 8 tasks
16/09/07 19:11:24 INFO TaskSetManager: Starting task 0.0 in stage 743.0 (TID 5944, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 1.0 in stage 743.0 (TID 5945, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 2.0 in stage 743.0 (TID 5946, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 3.0 in stage 743.0 (TID 5947, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 4.0 in stage 743.0 (TID 5948, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 5.0 in stage 743.0 (TID 5949, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 6.0 in stage 743.0 (TID 5950, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:24 INFO TaskSetManager: Starting task 7.0 in stage 743.0 (TID 5951, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:24 INFO Executor: Running task 0.0 in stage 743.0 (TID 5944)
16/09/07 19:11:24 INFO Executor: Running task 3.0 in stage 743.0 (TID 5947)
16/09/07 19:11:24 INFO Executor: Running task 1.0 in stage 743.0 (TID 5945)
16/09/07 19:11:24 INFO Executor: Running task 7.0 in stage 743.0 (TID 5951)
16/09/07 19:11:24 INFO Executor: Running task 2.0 in stage 743.0 (TID 5946)
16/09/07 19:11:24 INFO Executor: Running task 6.0 in stage 743.0 (TID 5950)
16/09/07 19:11:24 INFO Executor: Running task 4.0 in stage 743.0 (TID 5948)
16/09/07 19:11:24 INFO Executor: Running task 5.0 in stage 743.0 (TID 5949)
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:24 INFO Executor: Finished task 1.0 in stage 743.0 (TID 5945). 2160 bytes result sent to driver
16/09/07 19:11:24 INFO Executor: Finished task 4.0 in stage 743.0 (TID 5948). 2160 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 1.0 in stage 743.0 (TID 5945) in 2 ms on localhost (1/8)
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:24 INFO TaskSetManager: Finished task 4.0 in stage 743.0 (TID 5948) in 1 ms on localhost (2/8)
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:24 INFO Executor: Finished task 2.0 in stage 743.0 (TID 5946). 2160 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 2.0 in stage 743.0 (TID 5946) in 3 ms on localhost (3/8)
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:24 INFO Executor: Finished task 3.0 in stage 743.0 (TID 5947). 2160 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 3.0 in stage 743.0 (TID 5947) in 4 ms on localhost (4/8)
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:24 INFO Executor: Finished task 5.0 in stage 743.0 (TID 5949). 2160 bytes result sent to driver
16/09/07 19:11:24 INFO TaskSetManager: Finished task 5.0 in stage 743.0 (TID 5949) in 4 ms on localhost (5/8)
16/09/07 19:11:24 INFO BlockManagerInfo: Removed broadcast_1122_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:25 INFO Executor: Finished task 0.0 in stage 743.0 (TID 5944). 3485274 bytes result sent to driver
16/09/07 19:11:25 INFO TaskSetManager: Finished task 0.0 in stage 743.0 (TID 5944) in 914 ms on localhost (6/8)
16/09/07 19:11:25 INFO Executor: Finished task 6.0 in stage 743.0 (TID 5950). 3485274 bytes result sent to driver
16/09/07 19:11:25 INFO Executor: Finished task 7.0 in stage 743.0 (TID 5951). 3485274 bytes result sent to driver
16/09/07 19:11:25 INFO TaskSetManager: Finished task 6.0 in stage 743.0 (TID 5950) in 956 ms on localhost (7/8)
16/09/07 19:11:25 INFO TaskSetManager: Finished task 7.0 in stage 743.0 (TID 5951) in 968 ms on localhost (8/8)
16/09/07 19:11:25 INFO DAGScheduler: ResultStage 743 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.970 s
16/09/07 19:11:25 INFO TaskSchedulerImpl: Removed TaskSet 743.0, whose tasks have all completed, from pool 
16/09/07 19:11:25 INFO DAGScheduler: Job 376 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.977992 s
16/09/07 19:11:25 INFO ParameterAveragingTrainingMaster: Completed training of split 55 of 78
16/09/07 19:11:25 INFO ParameterAveragingTrainingMaster: Starting training of split 56 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:25 INFO BlockManagerInfo: Removed broadcast_1123_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:25 INFO ContextCleaner: Cleaned accumulator 744
16/09/07 19:11:25 INFO ContextCleaner: Cleaned accumulator 743
16/09/07 19:11:25 INFO BlockManagerInfo: Removed broadcast_1121_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:25 INFO ContextCleaner: Cleaned shuffle 366
16/09/07 19:11:25 INFO MemoryStore: ensureFreeSpace(18829056) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:25 INFO MemoryStore: Block broadcast_1124 stored as values in memory (estimated size 18.0 MB, free 3.7 GB)
16/09/07 19:11:25 INFO MemoryStore: ensureFreeSpace(167976) called with curMem=19372880, maxMem=4042889625
16/09/07 19:11:25 INFO MemoryStore: Block broadcast_1124_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:25 INFO BlockManagerInfo: Added broadcast_1124_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:25 INFO SparkContext: Created broadcast 1124 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:25 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:25 INFO DAGScheduler: Registering RDD 2243 (repartition at SparkUtils.java:190)
16/09/07 19:11:25 INFO DAGScheduler: Got job 377 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:25 INFO DAGScheduler: Final stage: ResultStage 745(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 744)
16/09/07 19:11:25 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 744)
16/09/07 19:11:25 INFO DAGScheduler: Submitting ShuffleMapStage 744 (MapPartitionsRDD[2243] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:25 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19540856, maxMem=4042889625
16/09/07 19:11:25 INFO MemoryStore: Block broadcast_1125 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:25 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19545320, maxMem=4042889625
16/09/07 19:11:25 INFO MemoryStore: Block broadcast_1125_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:25 INFO BlockManagerInfo: Added broadcast_1125_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:25 INFO SparkContext: Created broadcast 1125 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:25 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 744 (MapPartitionsRDD[2243] at repartition at SparkUtils.java:190)
16/09/07 19:11:25 INFO TaskSchedulerImpl: Adding task set 744.0 with 8 tasks
16/09/07 19:11:25 INFO TaskSetManager: Starting task 0.0 in stage 744.0 (TID 5952, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 1.0 in stage 744.0 (TID 5953, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 2.0 in stage 744.0 (TID 5954, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 3.0 in stage 744.0 (TID 5955, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 4.0 in stage 744.0 (TID 5956, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 5.0 in stage 744.0 (TID 5957, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 6.0 in stage 744.0 (TID 5958, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 7.0 in stage 744.0 (TID 5959, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:25 INFO Executor: Running task 3.0 in stage 744.0 (TID 5955)
16/09/07 19:11:25 INFO Executor: Running task 2.0 in stage 744.0 (TID 5954)
16/09/07 19:11:25 INFO Executor: Running task 5.0 in stage 744.0 (TID 5957)
16/09/07 19:11:25 INFO Executor: Running task 4.0 in stage 744.0 (TID 5956)
16/09/07 19:11:25 INFO Executor: Running task 1.0 in stage 744.0 (TID 5953)
16/09/07 19:11:25 INFO Executor: Running task 7.0 in stage 744.0 (TID 5959)
16/09/07 19:11:25 INFO Executor: Running task 0.0 in stage 744.0 (TID 5952)
16/09/07 19:11:25 INFO Executor: Running task 6.0 in stage 744.0 (TID 5958)
16/09/07 19:11:25 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:25 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:25 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:25 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:25 INFO Executor: Finished task 7.0 in stage 744.0 (TID 5959). 2260 bytes result sent to driver
16/09/07 19:11:25 INFO Executor: Finished task 2.0 in stage 744.0 (TID 5954). 2260 bytes result sent to driver
16/09/07 19:11:25 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:25 INFO Executor: Finished task 1.0 in stage 744.0 (TID 5953). 2260 bytes result sent to driver
16/09/07 19:11:25 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:25 INFO TaskSetManager: Finished task 7.0 in stage 744.0 (TID 5959) in 2 ms on localhost (1/8)
16/09/07 19:11:25 INFO TaskSetManager: Finished task 2.0 in stage 744.0 (TID 5954) in 2 ms on localhost (2/8)
16/09/07 19:11:25 INFO TaskSetManager: Finished task 1.0 in stage 744.0 (TID 5953) in 2 ms on localhost (3/8)
16/09/07 19:11:25 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:25 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:25 INFO Executor: Finished task 4.0 in stage 744.0 (TID 5956). 2260 bytes result sent to driver
16/09/07 19:11:25 INFO Executor: Finished task 3.0 in stage 744.0 (TID 5955). 2260 bytes result sent to driver
16/09/07 19:11:25 INFO Executor: Finished task 5.0 in stage 744.0 (TID 5957). 2260 bytes result sent to driver
16/09/07 19:11:25 INFO Executor: Finished task 6.0 in stage 744.0 (TID 5958). 2260 bytes result sent to driver
16/09/07 19:11:25 INFO TaskSetManager: Finished task 4.0 in stage 744.0 (TID 5956) in 3 ms on localhost (4/8)
16/09/07 19:11:25 INFO TaskSetManager: Finished task 5.0 in stage 744.0 (TID 5957) in 3 ms on localhost (5/8)
16/09/07 19:11:25 INFO TaskSetManager: Finished task 3.0 in stage 744.0 (TID 5955) in 3 ms on localhost (6/8)
16/09/07 19:11:25 INFO Executor: Finished task 0.0 in stage 744.0 (TID 5952). 2260 bytes result sent to driver
16/09/07 19:11:25 INFO TaskSetManager: Finished task 6.0 in stage 744.0 (TID 5958) in 3 ms on localhost (7/8)
16/09/07 19:11:25 INFO TaskSetManager: Finished task 0.0 in stage 744.0 (TID 5952) in 4 ms on localhost (8/8)
16/09/07 19:11:25 INFO TaskSchedulerImpl: Removed TaskSet 744.0, whose tasks have all completed, from pool 
16/09/07 19:11:25 INFO DAGScheduler: ShuffleMapStage 744 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:25 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:25 INFO DAGScheduler: running: Set()
16/09/07 19:11:25 INFO DAGScheduler: waiting: Set(ResultStage 745)
16/09/07 19:11:25 INFO DAGScheduler: failed: Set()
16/09/07 19:11:25 INFO DAGScheduler: Missing parents for ResultStage 745: List()
16/09/07 19:11:25 INFO DAGScheduler: Submitting ResultStage 745 (MapPartitionsRDD[2247] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:25 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19547884, maxMem=4042889625
16/09/07 19:11:25 INFO MemoryStore: Block broadcast_1126 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:25 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19552980, maxMem=4042889625
16/09/07 19:11:25 INFO MemoryStore: Block broadcast_1126_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:25 INFO BlockManagerInfo: Added broadcast_1126_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:25 INFO SparkContext: Created broadcast 1126 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:25 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 745 (MapPartitionsRDD[2247] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:25 INFO TaskSchedulerImpl: Adding task set 745.0 with 8 tasks
16/09/07 19:11:25 INFO TaskSetManager: Starting task 0.0 in stage 745.0 (TID 5960, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 1.0 in stage 745.0 (TID 5961, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 2.0 in stage 745.0 (TID 5962, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 3.0 in stage 745.0 (TID 5963, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 4.0 in stage 745.0 (TID 5964, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 5.0 in stage 745.0 (TID 5965, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 6.0 in stage 745.0 (TID 5966, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:25 INFO TaskSetManager: Starting task 7.0 in stage 745.0 (TID 5967, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:25 INFO Executor: Running task 0.0 in stage 745.0 (TID 5960)
16/09/07 19:11:25 INFO Executor: Running task 1.0 in stage 745.0 (TID 5961)
16/09/07 19:11:25 INFO Executor: Running task 4.0 in stage 745.0 (TID 5964)
16/09/07 19:11:25 INFO Executor: Running task 6.0 in stage 745.0 (TID 5966)
16/09/07 19:11:25 INFO Executor: Running task 5.0 in stage 745.0 (TID 5965)
16/09/07 19:11:25 INFO Executor: Running task 2.0 in stage 745.0 (TID 5962)
16/09/07 19:11:25 INFO Executor: Running task 7.0 in stage 745.0 (TID 5967)
16/09/07 19:11:25 INFO Executor: Running task 3.0 in stage 745.0 (TID 5963)
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:25 INFO Executor: Finished task 4.0 in stage 745.0 (TID 5964). 2160 bytes result sent to driver
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:25 INFO TaskSetManager: Finished task 4.0 in stage 745.0 (TID 5964) in 2 ms on localhost (1/8)
16/09/07 19:11:25 INFO Executor: Finished task 2.0 in stage 745.0 (TID 5962). 2160 bytes result sent to driver
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:25 INFO TaskSetManager: Finished task 2.0 in stage 745.0 (TID 5962) in 2 ms on localhost (2/8)
16/09/07 19:11:25 INFO Executor: Finished task 5.0 in stage 745.0 (TID 5965). 2160 bytes result sent to driver
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:25 INFO TaskSetManager: Finished task 5.0 in stage 745.0 (TID 5965) in 2 ms on localhost (3/8)
16/09/07 19:11:25 INFO Executor: Finished task 1.0 in stage 745.0 (TID 5961). 2160 bytes result sent to driver
16/09/07 19:11:25 INFO TaskSetManager: Finished task 1.0 in stage 745.0 (TID 5961) in 3 ms on localhost (4/8)
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:25 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:25 INFO Executor: Finished task 3.0 in stage 745.0 (TID 5963). 2160 bytes result sent to driver
16/09/07 19:11:25 INFO TaskSetManager: Finished task 3.0 in stage 745.0 (TID 5963) in 4 ms on localhost (5/8)
16/09/07 19:11:25 INFO BlockManagerInfo: Removed broadcast_1125_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:26 INFO Executor: Finished task 0.0 in stage 745.0 (TID 5960). 3485274 bytes result sent to driver
16/09/07 19:11:26 INFO Executor: Finished task 6.0 in stage 745.0 (TID 5966). 3485274 bytes result sent to driver
16/09/07 19:11:26 INFO Executor: Finished task 7.0 in stage 745.0 (TID 5967). 3485274 bytes result sent to driver
16/09/07 19:11:26 INFO TaskSetManager: Finished task 0.0 in stage 745.0 (TID 5960) in 890 ms on localhost (6/8)
16/09/07 19:11:26 INFO TaskSetManager: Finished task 6.0 in stage 745.0 (TID 5966) in 890 ms on localhost (7/8)
16/09/07 19:11:26 INFO TaskSetManager: Finished task 7.0 in stage 745.0 (TID 5967) in 890 ms on localhost (8/8)
16/09/07 19:11:26 INFO DAGScheduler: ResultStage 745 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.891 s
16/09/07 19:11:26 INFO TaskSchedulerImpl: Removed TaskSet 745.0, whose tasks have all completed, from pool 
16/09/07 19:11:26 INFO DAGScheduler: Job 377 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.898635 s
16/09/07 19:11:26 INFO ParameterAveragingTrainingMaster: Completed training of split 56 of 78
16/09/07 19:11:26 INFO ParameterAveragingTrainingMaster: Starting training of split 57 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:26 INFO MemoryStore: ensureFreeSpace(19017736) called with curMem=19548695, maxMem=4042889625
16/09/07 19:11:26 INFO MemoryStore: Block broadcast_1127 stored as values in memory (estimated size 18.1 MB, free 3.7 GB)
16/09/07 19:11:26 INFO MemoryStore: ensureFreeSpace(168007) called with curMem=38566431, maxMem=4042889625
16/09/07 19:11:26 INFO MemoryStore: Block broadcast_1127_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:26 INFO BlockManagerInfo: Added broadcast_1127_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:26 INFO SparkContext: Created broadcast 1127 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:26 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:26 INFO DAGScheduler: Registering RDD 2248 (repartition at SparkUtils.java:190)
16/09/07 19:11:26 INFO DAGScheduler: Got job 378 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:26 INFO DAGScheduler: Final stage: ResultStage 747(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:26 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 746)
16/09/07 19:11:26 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 746)
16/09/07 19:11:26 INFO DAGScheduler: Submitting ShuffleMapStage 746 (MapPartitionsRDD[2248] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:26 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=38734438, maxMem=4042889625
16/09/07 19:11:26 INFO MemoryStore: Block broadcast_1128 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:26 INFO MemoryStore: ensureFreeSpace(2560) called with curMem=38738902, maxMem=4042889625
16/09/07 19:11:26 INFO MemoryStore: Block broadcast_1128_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:26 INFO BlockManagerInfo: Added broadcast_1128_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:26 INFO SparkContext: Created broadcast 1128 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:26 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 746 (MapPartitionsRDD[2248] at repartition at SparkUtils.java:190)
16/09/07 19:11:26 INFO TaskSchedulerImpl: Adding task set 746.0 with 8 tasks
16/09/07 19:11:26 INFO TaskSetManager: Starting task 0.0 in stage 746.0 (TID 5968, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 1.0 in stage 746.0 (TID 5969, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 2.0 in stage 746.0 (TID 5970, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 3.0 in stage 746.0 (TID 5971, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 4.0 in stage 746.0 (TID 5972, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 5.0 in stage 746.0 (TID 5973, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 6.0 in stage 746.0 (TID 5974, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 7.0 in stage 746.0 (TID 5975, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:26 INFO Executor: Running task 2.0 in stage 746.0 (TID 5970)
16/09/07 19:11:26 INFO Executor: Running task 3.0 in stage 746.0 (TID 5971)
16/09/07 19:11:26 INFO Executor: Running task 5.0 in stage 746.0 (TID 5973)
16/09/07 19:11:26 INFO Executor: Running task 4.0 in stage 746.0 (TID 5972)
16/09/07 19:11:26 INFO Executor: Running task 6.0 in stage 746.0 (TID 5974)
16/09/07 19:11:26 INFO Executor: Running task 7.0 in stage 746.0 (TID 5975)
16/09/07 19:11:26 INFO Executor: Running task 0.0 in stage 746.0 (TID 5968)
16/09/07 19:11:26 INFO Executor: Running task 1.0 in stage 746.0 (TID 5969)
16/09/07 19:11:26 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:26 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:26 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:26 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:26 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:26 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:26 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:26 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:26 INFO Executor: Finished task 6.0 in stage 746.0 (TID 5974). 2260 bytes result sent to driver
16/09/07 19:11:26 INFO Executor: Finished task 0.0 in stage 746.0 (TID 5968). 2260 bytes result sent to driver
16/09/07 19:11:26 INFO Executor: Finished task 2.0 in stage 746.0 (TID 5970). 2260 bytes result sent to driver
16/09/07 19:11:26 INFO TaskSetManager: Finished task 6.0 in stage 746.0 (TID 5974) in 2 ms on localhost (1/8)
16/09/07 19:11:26 INFO TaskSetManager: Finished task 2.0 in stage 746.0 (TID 5970) in 3 ms on localhost (2/8)
16/09/07 19:11:26 INFO TaskSetManager: Finished task 0.0 in stage 746.0 (TID 5968) in 3 ms on localhost (3/8)
16/09/07 19:11:26 INFO Executor: Finished task 3.0 in stage 746.0 (TID 5971). 2260 bytes result sent to driver
16/09/07 19:11:26 INFO TaskSetManager: Finished task 3.0 in stage 746.0 (TID 5971) in 3 ms on localhost (4/8)
16/09/07 19:11:26 INFO Executor: Finished task 1.0 in stage 746.0 (TID 5969). 2260 bytes result sent to driver
16/09/07 19:11:26 INFO Executor: Finished task 5.0 in stage 746.0 (TID 5973). 2260 bytes result sent to driver
16/09/07 19:11:26 INFO Executor: Finished task 4.0 in stage 746.0 (TID 5972). 2260 bytes result sent to driver
16/09/07 19:11:26 INFO Executor: Finished task 7.0 in stage 746.0 (TID 5975). 2260 bytes result sent to driver
16/09/07 19:11:26 INFO TaskSetManager: Finished task 1.0 in stage 746.0 (TID 5969) in 3 ms on localhost (5/8)
16/09/07 19:11:26 INFO TaskSetManager: Finished task 7.0 in stage 746.0 (TID 5975) in 2 ms on localhost (6/8)
16/09/07 19:11:26 INFO TaskSetManager: Finished task 4.0 in stage 746.0 (TID 5972) in 3 ms on localhost (7/8)
16/09/07 19:11:26 INFO TaskSetManager: Finished task 5.0 in stage 746.0 (TID 5973) in 3 ms on localhost (8/8)
16/09/07 19:11:26 INFO TaskSchedulerImpl: Removed TaskSet 746.0, whose tasks have all completed, from pool 
16/09/07 19:11:26 INFO DAGScheduler: ShuffleMapStage 746 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:26 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:26 INFO DAGScheduler: running: Set()
16/09/07 19:11:26 INFO DAGScheduler: waiting: Set(ResultStage 747)
16/09/07 19:11:26 INFO DAGScheduler: failed: Set()
16/09/07 19:11:26 INFO DAGScheduler: Missing parents for ResultStage 747: List()
16/09/07 19:11:26 INFO DAGScheduler: Submitting ResultStage 747 (MapPartitionsRDD[2252] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:26 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=38741462, maxMem=4042889625
16/09/07 19:11:26 INFO MemoryStore: Block broadcast_1129 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:26 INFO BlockManagerInfo: Removed broadcast_1126_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:26 INFO ContextCleaner: Cleaned accumulator 746
16/09/07 19:11:26 INFO ContextCleaner: Cleaned accumulator 745
16/09/07 19:11:26 INFO BlockManagerInfo: Removed broadcast_1124_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:26 INFO MemoryStore: ensureFreeSpace(2742) called with curMem=19741687, maxMem=4042889625
16/09/07 19:11:26 INFO MemoryStore: Block broadcast_1129_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:26 INFO ContextCleaner: Cleaned shuffle 367
16/09/07 19:11:26 INFO BlockManagerInfo: Added broadcast_1129_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:26 INFO SparkContext: Created broadcast 1129 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:26 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 747 (MapPartitionsRDD[2252] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:26 INFO TaskSchedulerImpl: Adding task set 747.0 with 8 tasks
16/09/07 19:11:26 INFO TaskSetManager: Starting task 0.0 in stage 747.0 (TID 5976, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 1.0 in stage 747.0 (TID 5977, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 2.0 in stage 747.0 (TID 5978, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 3.0 in stage 747.0 (TID 5979, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 4.0 in stage 747.0 (TID 5980, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 5.0 in stage 747.0 (TID 5981, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 6.0 in stage 747.0 (TID 5982, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:26 INFO TaskSetManager: Starting task 7.0 in stage 747.0 (TID 5983, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:26 INFO Executor: Running task 0.0 in stage 747.0 (TID 5976)
16/09/07 19:11:26 INFO Executor: Running task 1.0 in stage 747.0 (TID 5977)
16/09/07 19:11:26 INFO Executor: Running task 5.0 in stage 747.0 (TID 5981)
16/09/07 19:11:26 INFO Executor: Running task 2.0 in stage 747.0 (TID 5978)
16/09/07 19:11:26 INFO Executor: Running task 4.0 in stage 747.0 (TID 5980)
16/09/07 19:11:26 INFO Executor: Running task 3.0 in stage 747.0 (TID 5979)
16/09/07 19:11:26 INFO Executor: Running task 7.0 in stage 747.0 (TID 5983)
16/09/07 19:11:26 INFO Executor: Running task 6.0 in stage 747.0 (TID 5982)
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:26 INFO Executor: Finished task 3.0 in stage 747.0 (TID 5979). 2160 bytes result sent to driver
16/09/07 19:11:26 INFO Executor: Finished task 4.0 in stage 747.0 (TID 5980). 2160 bytes result sent to driver
16/09/07 19:11:26 INFO TaskSetManager: Finished task 3.0 in stage 747.0 (TID 5979) in 2 ms on localhost (1/8)
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:26 INFO Executor: Finished task 5.0 in stage 747.0 (TID 5981). 2160 bytes result sent to driver
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:26 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:26 INFO TaskSetManager: Finished task 4.0 in stage 747.0 (TID 5980) in 3 ms on localhost (2/8)
16/09/07 19:11:26 INFO Executor: Finished task 2.0 in stage 747.0 (TID 5978). 2160 bytes result sent to driver
16/09/07 19:11:26 INFO TaskSetManager: Finished task 5.0 in stage 747.0 (TID 5981) in 3 ms on localhost (3/8)
16/09/07 19:11:26 INFO TaskSetManager: Finished task 2.0 in stage 747.0 (TID 5978) in 3 ms on localhost (4/8)
16/09/07 19:11:26 INFO Executor: Finished task 1.0 in stage 747.0 (TID 5977). 2160 bytes result sent to driver
16/09/07 19:11:26 INFO TaskSetManager: Finished task 1.0 in stage 747.0 (TID 5977) in 5 ms on localhost (5/8)
16/09/07 19:11:27 INFO BlockManagerInfo: Removed broadcast_1128_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:27 INFO Executor: Finished task 7.0 in stage 747.0 (TID 5983). 3485274 bytes result sent to driver
16/09/07 19:11:27 INFO Executor: Finished task 0.0 in stage 747.0 (TID 5976). 3485274 bytes result sent to driver
16/09/07 19:11:27 INFO Executor: Finished task 6.0 in stage 747.0 (TID 5982). 3485274 bytes result sent to driver
16/09/07 19:11:27 INFO TaskSetManager: Finished task 7.0 in stage 747.0 (TID 5983) in 796 ms on localhost (6/8)
16/09/07 19:11:27 INFO TaskSetManager: Finished task 0.0 in stage 747.0 (TID 5976) in 798 ms on localhost (7/8)
16/09/07 19:11:27 INFO TaskSetManager: Finished task 6.0 in stage 747.0 (TID 5982) in 799 ms on localhost (8/8)
16/09/07 19:11:27 INFO TaskSchedulerImpl: Removed TaskSet 747.0, whose tasks have all completed, from pool 
16/09/07 19:11:27 INFO DAGScheduler: ResultStage 747 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.800 s
16/09/07 19:11:27 INFO DAGScheduler: Job 378 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.881092 s
16/09/07 19:11:27 INFO ParameterAveragingTrainingMaster: Completed training of split 57 of 78
16/09/07 19:11:27 INFO ParameterAveragingTrainingMaster: Starting training of split 58 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:27 INFO MemoryStore: ensureFreeSpace(19036584) called with curMem=19737405, maxMem=4042889625
16/09/07 19:11:27 INFO MemoryStore: Block broadcast_1130 stored as values in memory (estimated size 18.2 MB, free 3.7 GB)
16/09/07 19:11:27 INFO MemoryStore: ensureFreeSpace(167953) called with curMem=38773989, maxMem=4042889625
16/09/07 19:11:27 INFO MemoryStore: Block broadcast_1130_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:27 INFO BlockManagerInfo: Added broadcast_1130_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:27 INFO SparkContext: Created broadcast 1130 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:28 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:28 INFO DAGScheduler: Registering RDD 2253 (repartition at SparkUtils.java:190)
16/09/07 19:11:28 INFO DAGScheduler: Got job 379 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:28 INFO DAGScheduler: Final stage: ResultStage 749(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:28 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 748)
16/09/07 19:11:28 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 748)
16/09/07 19:11:28 INFO DAGScheduler: Submitting ShuffleMapStage 748 (MapPartitionsRDD[2253] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:28 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=38941942, maxMem=4042889625
16/09/07 19:11:28 INFO MemoryStore: Block broadcast_1131 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:28 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=38946406, maxMem=4042889625
16/09/07 19:11:28 INFO MemoryStore: Block broadcast_1131_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:28 INFO BlockManagerInfo: Added broadcast_1131_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:28 INFO SparkContext: Created broadcast 1131 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:28 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 748 (MapPartitionsRDD[2253] at repartition at SparkUtils.java:190)
16/09/07 19:11:28 INFO TaskSchedulerImpl: Adding task set 748.0 with 8 tasks
16/09/07 19:11:28 INFO TaskSetManager: Starting task 0.0 in stage 748.0 (TID 5984, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 1.0 in stage 748.0 (TID 5985, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 2.0 in stage 748.0 (TID 5986, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 3.0 in stage 748.0 (TID 5987, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 4.0 in stage 748.0 (TID 5988, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 5.0 in stage 748.0 (TID 5989, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 6.0 in stage 748.0 (TID 5990, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 7.0 in stage 748.0 (TID 5991, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:28 INFO Executor: Running task 0.0 in stage 748.0 (TID 5984)
16/09/07 19:11:28 INFO Executor: Running task 3.0 in stage 748.0 (TID 5987)
16/09/07 19:11:28 INFO Executor: Running task 5.0 in stage 748.0 (TID 5989)
16/09/07 19:11:28 INFO Executor: Running task 6.0 in stage 748.0 (TID 5990)
16/09/07 19:11:28 INFO Executor: Running task 2.0 in stage 748.0 (TID 5986)
16/09/07 19:11:28 INFO Executor: Running task 7.0 in stage 748.0 (TID 5991)
16/09/07 19:11:28 INFO Executor: Running task 1.0 in stage 748.0 (TID 5985)
16/09/07 19:11:28 INFO Executor: Running task 4.0 in stage 748.0 (TID 5988)
16/09/07 19:11:28 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:28 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:28 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:28 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:28 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:28 INFO Executor: Finished task 4.0 in stage 748.0 (TID 5988). 2260 bytes result sent to driver
16/09/07 19:11:28 INFO Executor: Finished task 5.0 in stage 748.0 (TID 5989). 2260 bytes result sent to driver
16/09/07 19:11:28 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:28 INFO TaskSetManager: Finished task 4.0 in stage 748.0 (TID 5988) in 2 ms on localhost (1/8)
16/09/07 19:11:28 INFO Executor: Finished task 6.0 in stage 748.0 (TID 5990). 2260 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 5.0 in stage 748.0 (TID 5989) in 2 ms on localhost (2/8)
16/09/07 19:11:28 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:28 INFO TaskSetManager: Finished task 6.0 in stage 748.0 (TID 5990) in 3 ms on localhost (3/8)
16/09/07 19:11:28 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:28 INFO Executor: Finished task 7.0 in stage 748.0 (TID 5991). 2260 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 7.0 in stage 748.0 (TID 5991) in 3 ms on localhost (4/8)
16/09/07 19:11:28 INFO Executor: Finished task 1.0 in stage 748.0 (TID 5985). 2260 bytes result sent to driver
16/09/07 19:11:28 INFO Executor: Finished task 2.0 in stage 748.0 (TID 5986). 2260 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 1.0 in stage 748.0 (TID 5985) in 3 ms on localhost (5/8)
16/09/07 19:11:28 INFO Executor: Finished task 3.0 in stage 748.0 (TID 5987). 2260 bytes result sent to driver
16/09/07 19:11:28 INFO Executor: Finished task 0.0 in stage 748.0 (TID 5984). 2260 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 2.0 in stage 748.0 (TID 5986) in 3 ms on localhost (6/8)
16/09/07 19:11:28 INFO TaskSetManager: Finished task 3.0 in stage 748.0 (TID 5987) in 4 ms on localhost (7/8)
16/09/07 19:11:28 INFO TaskSetManager: Finished task 0.0 in stage 748.0 (TID 5984) in 4 ms on localhost (8/8)
16/09/07 19:11:28 INFO TaskSchedulerImpl: Removed TaskSet 748.0, whose tasks have all completed, from pool 
16/09/07 19:11:28 INFO DAGScheduler: ShuffleMapStage 748 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:28 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:28 INFO DAGScheduler: running: Set()
16/09/07 19:11:28 INFO DAGScheduler: waiting: Set(ResultStage 749)
16/09/07 19:11:28 INFO DAGScheduler: failed: Set()
16/09/07 19:11:28 INFO DAGScheduler: Missing parents for ResultStage 749: List()
16/09/07 19:11:28 INFO DAGScheduler: Submitting ResultStage 749 (MapPartitionsRDD[2257] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:28 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=38948970, maxMem=4042889625
16/09/07 19:11:28 INFO MemoryStore: Block broadcast_1132 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:28 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=38954066, maxMem=4042889625
16/09/07 19:11:28 INFO MemoryStore: Block broadcast_1132_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:28 INFO BlockManagerInfo: Added broadcast_1132_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:28 INFO SparkContext: Created broadcast 1132 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:28 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 749 (MapPartitionsRDD[2257] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:28 INFO TaskSchedulerImpl: Adding task set 749.0 with 8 tasks
16/09/07 19:11:28 INFO TaskSetManager: Starting task 0.0 in stage 749.0 (TID 5992, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 1.0 in stage 749.0 (TID 5993, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 2.0 in stage 749.0 (TID 5994, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 3.0 in stage 749.0 (TID 5995, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 4.0 in stage 749.0 (TID 5996, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 5.0 in stage 749.0 (TID 5997, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 6.0 in stage 749.0 (TID 5998, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:28 INFO TaskSetManager: Starting task 7.0 in stage 749.0 (TID 5999, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:28 INFO Executor: Running task 1.0 in stage 749.0 (TID 5993)
16/09/07 19:11:28 INFO Executor: Running task 6.0 in stage 749.0 (TID 5998)
16/09/07 19:11:28 INFO Executor: Running task 7.0 in stage 749.0 (TID 5999)
16/09/07 19:11:28 INFO Executor: Running task 3.0 in stage 749.0 (TID 5995)
16/09/07 19:11:28 INFO Executor: Running task 0.0 in stage 749.0 (TID 5992)
16/09/07 19:11:28 INFO Executor: Running task 4.0 in stage 749.0 (TID 5996)
16/09/07 19:11:28 INFO Executor: Running task 2.0 in stage 749.0 (TID 5994)
16/09/07 19:11:28 INFO Executor: Running task 5.0 in stage 749.0 (TID 5997)
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:28 INFO BlockManagerInfo: Removed broadcast_1129_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:28 INFO Executor: Finished task 5.0 in stage 749.0 (TID 5997). 2160 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 5.0 in stage 749.0 (TID 5997) in 79 ms on localhost (1/8)
16/09/07 19:11:28 INFO BlockManagerInfo: Removed broadcast_1131_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:28 INFO Executor: Finished task 1.0 in stage 749.0 (TID 5993). 2160 bytes result sent to driver
16/09/07 19:11:28 INFO ContextCleaner: Cleaned accumulator 748
16/09/07 19:11:28 INFO ContextCleaner: Cleaned accumulator 747
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:28 INFO TaskSetManager: Finished task 1.0 in stage 749.0 (TID 5993) in 79 ms on localhost (2/8)
16/09/07 19:11:28 INFO BlockManagerInfo: Removed broadcast_1127_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:28 INFO Executor: Finished task 4.0 in stage 749.0 (TID 5996). 2160 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 4.0 in stage 749.0 (TID 5996) in 80 ms on localhost (3/8)
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:28 INFO Executor: Finished task 3.0 in stage 749.0 (TID 5995). 2160 bytes result sent to driver
16/09/07 19:11:28 INFO ContextCleaner: Cleaned shuffle 368
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:28 INFO TaskSetManager: Finished task 3.0 in stage 749.0 (TID 5995) in 80 ms on localhost (4/8)
16/09/07 19:11:28 INFO Executor: Finished task 2.0 in stage 749.0 (TID 5994). 2160 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 2.0 in stage 749.0 (TID 5994) in 81 ms on localhost (5/8)
16/09/07 19:11:28 INFO Executor: Finished task 7.0 in stage 749.0 (TID 5999). 3485274 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 7.0 in stage 749.0 (TID 5999) in 854 ms on localhost (6/8)
16/09/07 19:11:28 INFO Executor: Finished task 6.0 in stage 749.0 (TID 5998). 3485274 bytes result sent to driver
16/09/07 19:11:28 INFO Executor: Finished task 0.0 in stage 749.0 (TID 5992). 3485274 bytes result sent to driver
16/09/07 19:11:28 INFO TaskSetManager: Finished task 6.0 in stage 749.0 (TID 5998) in 901 ms on localhost (7/8)
16/09/07 19:11:28 INFO TaskSetManager: Finished task 0.0 in stage 749.0 (TID 5992) in 915 ms on localhost (8/8)
16/09/07 19:11:28 INFO TaskSchedulerImpl: Removed TaskSet 749.0, whose tasks have all completed, from pool 
16/09/07 19:11:28 INFO DAGScheduler: ResultStage 749 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.915 s
16/09/07 19:11:28 INFO DAGScheduler: Job 379 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.922783 s
16/09/07 19:11:28 INFO ParameterAveragingTrainingMaster: Completed training of split 58 of 78
16/09/07 19:11:28 INFO ParameterAveragingTrainingMaster: Starting training of split 59 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:29 INFO BlockManagerInfo: Removed broadcast_1132_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:29 INFO ContextCleaner: Cleaned accumulator 750
16/09/07 19:11:29 INFO ContextCleaner: Cleaned accumulator 749
16/09/07 19:11:29 INFO BlockManagerInfo: Removed broadcast_1130_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:29 INFO ContextCleaner: Cleaned shuffle 369
16/09/07 19:11:29 INFO MemoryStore: ensureFreeSpace(18914968) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:29 INFO MemoryStore: Block broadcast_1133 stored as values in memory (estimated size 18.0 MB, free 3.7 GB)
16/09/07 19:11:29 INFO MemoryStore: ensureFreeSpace(167931) called with curMem=19458792, maxMem=4042889625
16/09/07 19:11:29 INFO MemoryStore: Block broadcast_1133_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:29 INFO BlockManagerInfo: Added broadcast_1133_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:29 INFO SparkContext: Created broadcast 1133 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:29 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:29 INFO DAGScheduler: Registering RDD 2258 (repartition at SparkUtils.java:190)
16/09/07 19:11:29 INFO DAGScheduler: Got job 380 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:29 INFO DAGScheduler: Final stage: ResultStage 751(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:29 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 750)
16/09/07 19:11:29 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 750)
16/09/07 19:11:29 INFO DAGScheduler: Submitting ShuffleMapStage 750 (MapPartitionsRDD[2258] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:29 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19626723, maxMem=4042889625
16/09/07 19:11:29 INFO MemoryStore: Block broadcast_1134 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:29 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=19631187, maxMem=4042889625
16/09/07 19:11:29 INFO MemoryStore: Block broadcast_1134_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:29 INFO BlockManagerInfo: Added broadcast_1134_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:29 INFO SparkContext: Created broadcast 1134 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:29 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 750 (MapPartitionsRDD[2258] at repartition at SparkUtils.java:190)
16/09/07 19:11:29 INFO TaskSchedulerImpl: Adding task set 750.0 with 8 tasks
16/09/07 19:11:29 INFO TaskSetManager: Starting task 0.0 in stage 750.0 (TID 6000, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 1.0 in stage 750.0 (TID 6001, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 2.0 in stage 750.0 (TID 6002, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 3.0 in stage 750.0 (TID 6003, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 4.0 in stage 750.0 (TID 6004, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 5.0 in stage 750.0 (TID 6005, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 6.0 in stage 750.0 (TID 6006, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 7.0 in stage 750.0 (TID 6007, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:29 INFO Executor: Running task 0.0 in stage 750.0 (TID 6000)
16/09/07 19:11:29 INFO Executor: Running task 1.0 in stage 750.0 (TID 6001)
16/09/07 19:11:29 INFO Executor: Running task 2.0 in stage 750.0 (TID 6002)
16/09/07 19:11:29 INFO Executor: Running task 7.0 in stage 750.0 (TID 6007)
16/09/07 19:11:29 INFO Executor: Running task 3.0 in stage 750.0 (TID 6003)
16/09/07 19:11:29 INFO Executor: Running task 4.0 in stage 750.0 (TID 6004)
16/09/07 19:11:29 INFO Executor: Running task 5.0 in stage 750.0 (TID 6005)
16/09/07 19:11:29 INFO Executor: Running task 6.0 in stage 750.0 (TID 6006)
16/09/07 19:11:29 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:29 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:29 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:29 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:29 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:29 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:29 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:29 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:29 INFO Executor: Finished task 1.0 in stage 750.0 (TID 6001). 2260 bytes result sent to driver
16/09/07 19:11:29 INFO TaskSetManager: Finished task 1.0 in stage 750.0 (TID 6001) in 3 ms on localhost (1/8)
16/09/07 19:11:29 INFO Executor: Finished task 7.0 in stage 750.0 (TID 6007). 2260 bytes result sent to driver
16/09/07 19:11:29 INFO Executor: Finished task 2.0 in stage 750.0 (TID 6002). 2260 bytes result sent to driver
16/09/07 19:11:29 INFO TaskSetManager: Finished task 7.0 in stage 750.0 (TID 6007) in 3 ms on localhost (2/8)
16/09/07 19:11:29 INFO TaskSetManager: Finished task 2.0 in stage 750.0 (TID 6002) in 3 ms on localhost (3/8)
16/09/07 19:11:29 INFO Executor: Finished task 6.0 in stage 750.0 (TID 6006). 2260 bytes result sent to driver
16/09/07 19:11:29 INFO Executor: Finished task 3.0 in stage 750.0 (TID 6003). 2260 bytes result sent to driver
16/09/07 19:11:29 INFO Executor: Finished task 5.0 in stage 750.0 (TID 6005). 2260 bytes result sent to driver
16/09/07 19:11:29 INFO TaskSetManager: Finished task 6.0 in stage 750.0 (TID 6006) in 4 ms on localhost (4/8)
16/09/07 19:11:29 INFO Executor: Finished task 0.0 in stage 750.0 (TID 6000). 2260 bytes result sent to driver
16/09/07 19:11:29 INFO TaskSetManager: Finished task 3.0 in stage 750.0 (TID 6003) in 4 ms on localhost (5/8)
16/09/07 19:11:29 INFO Executor: Finished task 4.0 in stage 750.0 (TID 6004). 2260 bytes result sent to driver
16/09/07 19:11:29 INFO TaskSetManager: Finished task 5.0 in stage 750.0 (TID 6005) in 4 ms on localhost (6/8)
16/09/07 19:11:29 INFO TaskSetManager: Finished task 4.0 in stage 750.0 (TID 6004) in 4 ms on localhost (7/8)
16/09/07 19:11:29 INFO TaskSetManager: Finished task 0.0 in stage 750.0 (TID 6000) in 4 ms on localhost (8/8)
16/09/07 19:11:29 INFO DAGScheduler: ShuffleMapStage 750 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:29 INFO TaskSchedulerImpl: Removed TaskSet 750.0, whose tasks have all completed, from pool 
16/09/07 19:11:29 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:29 INFO DAGScheduler: running: Set()
16/09/07 19:11:29 INFO DAGScheduler: waiting: Set(ResultStage 751)
16/09/07 19:11:29 INFO DAGScheduler: failed: Set()
16/09/07 19:11:29 INFO DAGScheduler: Missing parents for ResultStage 751: List()
16/09/07 19:11:29 INFO DAGScheduler: Submitting ResultStage 751 (MapPartitionsRDD[2262] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:29 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19633749, maxMem=4042889625
16/09/07 19:11:29 INFO MemoryStore: Block broadcast_1135 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:29 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=19638845, maxMem=4042889625
16/09/07 19:11:29 INFO MemoryStore: Block broadcast_1135_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:29 INFO BlockManagerInfo: Added broadcast_1135_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:29 INFO SparkContext: Created broadcast 1135 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:29 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 751 (MapPartitionsRDD[2262] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:29 INFO TaskSchedulerImpl: Adding task set 751.0 with 8 tasks
16/09/07 19:11:29 INFO TaskSetManager: Starting task 0.0 in stage 751.0 (TID 6008, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 1.0 in stage 751.0 (TID 6009, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 2.0 in stage 751.0 (TID 6010, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 3.0 in stage 751.0 (TID 6011, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 4.0 in stage 751.0 (TID 6012, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 5.0 in stage 751.0 (TID 6013, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 6.0 in stage 751.0 (TID 6014, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:29 INFO TaskSetManager: Starting task 7.0 in stage 751.0 (TID 6015, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:29 INFO Executor: Running task 0.0 in stage 751.0 (TID 6008)
16/09/07 19:11:29 INFO Executor: Running task 7.0 in stage 751.0 (TID 6015)
16/09/07 19:11:29 INFO Executor: Running task 6.0 in stage 751.0 (TID 6014)
16/09/07 19:11:29 INFO Executor: Running task 5.0 in stage 751.0 (TID 6013)
16/09/07 19:11:29 INFO Executor: Running task 4.0 in stage 751.0 (TID 6012)
16/09/07 19:11:29 INFO Executor: Running task 1.0 in stage 751.0 (TID 6009)
16/09/07 19:11:29 INFO Executor: Running task 3.0 in stage 751.0 (TID 6011)
16/09/07 19:11:29 INFO Executor: Running task 2.0 in stage 751.0 (TID 6010)
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:29 INFO Executor: Finished task 2.0 in stage 751.0 (TID 6010). 2160 bytes result sent to driver
16/09/07 19:11:29 INFO Executor: Finished task 1.0 in stage 751.0 (TID 6009). 2160 bytes result sent to driver
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:29 INFO Executor: Finished task 3.0 in stage 751.0 (TID 6011). 2160 bytes result sent to driver
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:29 INFO Executor: Finished task 5.0 in stage 751.0 (TID 6013). 2160 bytes result sent to driver
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:29 INFO Executor: Finished task 4.0 in stage 751.0 (TID 6012). 2160 bytes result sent to driver
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:29 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:29 INFO TaskSetManager: Finished task 2.0 in stage 751.0 (TID 6010) in 2 ms on localhost (1/8)
16/09/07 19:11:29 INFO TaskSetManager: Finished task 1.0 in stage 751.0 (TID 6009) in 3 ms on localhost (2/8)
16/09/07 19:11:29 INFO TaskSetManager: Finished task 3.0 in stage 751.0 (TID 6011) in 3 ms on localhost (3/8)
16/09/07 19:11:29 INFO TaskSetManager: Finished task 5.0 in stage 751.0 (TID 6013) in 3 ms on localhost (4/8)
16/09/07 19:11:29 INFO TaskSetManager: Finished task 4.0 in stage 751.0 (TID 6012) in 3 ms on localhost (5/8)
16/09/07 19:11:29 INFO BlockManagerInfo: Removed broadcast_1134_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:30 INFO Executor: Finished task 0.0 in stage 751.0 (TID 6008). 3485274 bytes result sent to driver
16/09/07 19:11:30 INFO Executor: Finished task 7.0 in stage 751.0 (TID 6015). 3485274 bytes result sent to driver
16/09/07 19:11:30 INFO Executor: Finished task 6.0 in stage 751.0 (TID 6014). 3485274 bytes result sent to driver
16/09/07 19:11:30 INFO TaskSetManager: Finished task 0.0 in stage 751.0 (TID 6008) in 894 ms on localhost (6/8)
16/09/07 19:11:30 INFO TaskSetManager: Finished task 7.0 in stage 751.0 (TID 6015) in 895 ms on localhost (7/8)
16/09/07 19:11:30 INFO TaskSetManager: Finished task 6.0 in stage 751.0 (TID 6014) in 895 ms on localhost (8/8)
16/09/07 19:11:30 INFO TaskSchedulerImpl: Removed TaskSet 751.0, whose tasks have all completed, from pool 
16/09/07 19:11:30 INFO DAGScheduler: ResultStage 751 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.895 s
16/09/07 19:11:30 INFO DAGScheduler: Job 380 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.903465 s
16/09/07 19:11:30 INFO ParameterAveragingTrainingMaster: Completed training of split 59 of 78
16/09/07 19:11:30 INFO ParameterAveragingTrainingMaster: Starting training of split 60 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:30 INFO MemoryStore: ensureFreeSpace(19116360) called with curMem=19634563, maxMem=4042889625
16/09/07 19:11:30 INFO MemoryStore: Block broadcast_1136 stored as values in memory (estimated size 18.2 MB, free 3.7 GB)
16/09/07 19:11:30 INFO MemoryStore: ensureFreeSpace(167951) called with curMem=38750923, maxMem=4042889625
16/09/07 19:11:30 INFO MemoryStore: Block broadcast_1136_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:30 INFO BlockManagerInfo: Added broadcast_1136_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:30 INFO SparkContext: Created broadcast 1136 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:30 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:30 INFO DAGScheduler: Registering RDD 2263 (repartition at SparkUtils.java:190)
16/09/07 19:11:30 INFO DAGScheduler: Got job 381 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:30 INFO DAGScheduler: Final stage: ResultStage 753(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:30 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 752)
16/09/07 19:11:30 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 752)
16/09/07 19:11:30 INFO DAGScheduler: Submitting ShuffleMapStage 752 (MapPartitionsRDD[2263] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:30 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=38918874, maxMem=4042889625
16/09/07 19:11:30 INFO MemoryStore: Block broadcast_1137 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:30 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=38923338, maxMem=4042889625
16/09/07 19:11:30 INFO MemoryStore: Block broadcast_1137_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:30 INFO BlockManagerInfo: Added broadcast_1137_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:30 INFO SparkContext: Created broadcast 1137 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:30 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 752 (MapPartitionsRDD[2263] at repartition at SparkUtils.java:190)
16/09/07 19:11:30 INFO TaskSchedulerImpl: Adding task set 752.0 with 8 tasks
16/09/07 19:11:30 INFO TaskSetManager: Starting task 0.0 in stage 752.0 (TID 6016, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 1.0 in stage 752.0 (TID 6017, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 2.0 in stage 752.0 (TID 6018, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 3.0 in stage 752.0 (TID 6019, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 4.0 in stage 752.0 (TID 6020, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 5.0 in stage 752.0 (TID 6021, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 6.0 in stage 752.0 (TID 6022, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 7.0 in stage 752.0 (TID 6023, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:30 INFO Executor: Running task 3.0 in stage 752.0 (TID 6019)
16/09/07 19:11:30 INFO Executor: Running task 4.0 in stage 752.0 (TID 6020)
16/09/07 19:11:30 INFO Executor: Running task 5.0 in stage 752.0 (TID 6021)
16/09/07 19:11:30 INFO Executor: Running task 1.0 in stage 752.0 (TID 6017)
16/09/07 19:11:30 INFO Executor: Running task 0.0 in stage 752.0 (TID 6016)
16/09/07 19:11:30 INFO Executor: Running task 2.0 in stage 752.0 (TID 6018)
16/09/07 19:11:30 INFO Executor: Running task 6.0 in stage 752.0 (TID 6022)
16/09/07 19:11:30 INFO Executor: Running task 7.0 in stage 752.0 (TID 6023)
16/09/07 19:11:30 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:30 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:30 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:30 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:30 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:30 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:30 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:30 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:30 INFO Executor: Finished task 5.0 in stage 752.0 (TID 6021). 2260 bytes result sent to driver
16/09/07 19:11:30 INFO Executor: Finished task 1.0 in stage 752.0 (TID 6017). 2260 bytes result sent to driver
16/09/07 19:11:30 INFO TaskSetManager: Finished task 5.0 in stage 752.0 (TID 6021) in 3 ms on localhost (1/8)
16/09/07 19:11:30 INFO Executor: Finished task 0.0 in stage 752.0 (TID 6016). 2260 bytes result sent to driver
16/09/07 19:11:30 INFO TaskSetManager: Finished task 1.0 in stage 752.0 (TID 6017) in 3 ms on localhost (2/8)
16/09/07 19:11:30 INFO TaskSetManager: Finished task 0.0 in stage 752.0 (TID 6016) in 3 ms on localhost (3/8)
16/09/07 19:11:30 INFO Executor: Finished task 6.0 in stage 752.0 (TID 6022). 2260 bytes result sent to driver
16/09/07 19:11:30 INFO Executor: Finished task 7.0 in stage 752.0 (TID 6023). 2260 bytes result sent to driver
16/09/07 19:11:30 INFO Executor: Finished task 3.0 in stage 752.0 (TID 6019). 2260 bytes result sent to driver
16/09/07 19:11:30 INFO Executor: Finished task 2.0 in stage 752.0 (TID 6018). 2260 bytes result sent to driver
16/09/07 19:11:30 INFO Executor: Finished task 4.0 in stage 752.0 (TID 6020). 2260 bytes result sent to driver
16/09/07 19:11:30 INFO TaskSetManager: Finished task 6.0 in stage 752.0 (TID 6022) in 4 ms on localhost (4/8)
16/09/07 19:11:30 INFO TaskSetManager: Finished task 7.0 in stage 752.0 (TID 6023) in 4 ms on localhost (5/8)
16/09/07 19:11:30 INFO TaskSetManager: Finished task 3.0 in stage 752.0 (TID 6019) in 4 ms on localhost (6/8)
16/09/07 19:11:30 INFO TaskSetManager: Finished task 2.0 in stage 752.0 (TID 6018) in 4 ms on localhost (7/8)
16/09/07 19:11:30 INFO TaskSetManager: Finished task 4.0 in stage 752.0 (TID 6020) in 4 ms on localhost (8/8)
16/09/07 19:11:30 INFO TaskSchedulerImpl: Removed TaskSet 752.0, whose tasks have all completed, from pool 
16/09/07 19:11:30 INFO DAGScheduler: ShuffleMapStage 752 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:30 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:30 INFO DAGScheduler: running: Set()
16/09/07 19:11:30 INFO DAGScheduler: waiting: Set(ResultStage 753)
16/09/07 19:11:30 INFO DAGScheduler: failed: Set()
16/09/07 19:11:30 INFO DAGScheduler: Missing parents for ResultStage 753: List()
16/09/07 19:11:30 INFO DAGScheduler: Submitting ResultStage 753 (MapPartitionsRDD[2267] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:30 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=38925900, maxMem=4042889625
16/09/07 19:11:30 INFO MemoryStore: Block broadcast_1138 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:30 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=38930996, maxMem=4042889625
16/09/07 19:11:30 INFO MemoryStore: Block broadcast_1138_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:30 INFO BlockManagerInfo: Added broadcast_1138_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:30 INFO SparkContext: Created broadcast 1138 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:30 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 753 (MapPartitionsRDD[2267] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:30 INFO TaskSchedulerImpl: Adding task set 753.0 with 8 tasks
16/09/07 19:11:30 INFO TaskSetManager: Starting task 0.0 in stage 753.0 (TID 6024, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 1.0 in stage 753.0 (TID 6025, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 2.0 in stage 753.0 (TID 6026, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 3.0 in stage 753.0 (TID 6027, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 4.0 in stage 753.0 (TID 6028, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 5.0 in stage 753.0 (TID 6029, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 6.0 in stage 753.0 (TID 6030, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:30 INFO TaskSetManager: Starting task 7.0 in stage 753.0 (TID 6031, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:30 INFO Executor: Running task 0.0 in stage 753.0 (TID 6024)
16/09/07 19:11:30 INFO Executor: Running task 4.0 in stage 753.0 (TID 6028)
16/09/07 19:11:30 INFO Executor: Running task 6.0 in stage 753.0 (TID 6030)
16/09/07 19:11:30 INFO Executor: Running task 1.0 in stage 753.0 (TID 6025)
16/09/07 19:11:30 INFO Executor: Running task 2.0 in stage 753.0 (TID 6026)
16/09/07 19:11:30 INFO Executor: Running task 3.0 in stage 753.0 (TID 6027)
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:30 INFO Executor: Running task 5.0 in stage 753.0 (TID 6029)
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:30 INFO Executor: Running task 7.0 in stage 753.0 (TID 6031)
16/09/07 19:11:30 INFO Executor: Finished task 2.0 in stage 753.0 (TID 6026). 2160 bytes result sent to driver
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:30 INFO Executor: Finished task 1.0 in stage 753.0 (TID 6025). 2160 bytes result sent to driver
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:30 INFO TaskSetManager: Finished task 2.0 in stage 753.0 (TID 6026) in 2 ms on localhost (1/8)
16/09/07 19:11:30 INFO TaskSetManager: Finished task 1.0 in stage 753.0 (TID 6025) in 2 ms on localhost (2/8)
16/09/07 19:11:30 INFO Executor: Finished task 4.0 in stage 753.0 (TID 6028). 2160 bytes result sent to driver
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:30 INFO TaskSetManager: Finished task 4.0 in stage 753.0 (TID 6028) in 2 ms on localhost (3/8)
16/09/07 19:11:30 INFO Executor: Finished task 3.0 in stage 753.0 (TID 6027). 2160 bytes result sent to driver
16/09/07 19:11:30 INFO TaskSetManager: Finished task 3.0 in stage 753.0 (TID 6027) in 3 ms on localhost (4/8)
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:30 INFO Executor: Finished task 5.0 in stage 753.0 (TID 6029). 2160 bytes result sent to driver
16/09/07 19:11:30 INFO TaskSetManager: Finished task 5.0 in stage 753.0 (TID 6029) in 90 ms on localhost (5/8)
16/09/07 19:11:30 INFO BlockManagerInfo: Removed broadcast_1135_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:30 INFO BlockManagerInfo: Removed broadcast_1137_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:30 INFO ContextCleaner: Cleaned accumulator 752
16/09/07 19:11:30 INFO ContextCleaner: Cleaned accumulator 751
16/09/07 19:11:30 INFO BlockManagerInfo: Removed broadcast_1133_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:30 INFO ContextCleaner: Cleaned shuffle 370
16/09/07 19:11:31 INFO Executor: Finished task 7.0 in stage 753.0 (TID 6031). 3485274 bytes result sent to driver
16/09/07 19:11:31 INFO TaskSetManager: Finished task 7.0 in stage 753.0 (TID 6031) in 885 ms on localhost (6/8)
16/09/07 19:11:31 INFO Executor: Finished task 0.0 in stage 753.0 (TID 6024). 3485274 bytes result sent to driver
16/09/07 19:11:31 INFO Executor: Finished task 6.0 in stage 753.0 (TID 6030). 3485274 bytes result sent to driver
16/09/07 19:11:31 INFO TaskSetManager: Finished task 0.0 in stage 753.0 (TID 6024) in 932 ms on localhost (7/8)
16/09/07 19:11:31 INFO TaskSetManager: Finished task 6.0 in stage 753.0 (TID 6030) in 940 ms on localhost (8/8)
16/09/07 19:11:31 INFO TaskSchedulerImpl: Removed TaskSet 753.0, whose tasks have all completed, from pool 
16/09/07 19:11:31 INFO DAGScheduler: ResultStage 753 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.942 s
16/09/07 19:11:31 INFO DAGScheduler: Job 381 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.948800 s
16/09/07 19:11:31 INFO ParameterAveragingTrainingMaster: Completed training of split 60 of 78
16/09/07 19:11:31 INFO ParameterAveragingTrainingMaster: Starting training of split 61 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:31 INFO BlockManagerInfo: Removed broadcast_1138_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:31 INFO ContextCleaner: Cleaned accumulator 754
16/09/07 19:11:31 INFO ContextCleaner: Cleaned accumulator 753
16/09/07 19:11:31 INFO BlockManagerInfo: Removed broadcast_1136_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:31 INFO ContextCleaner: Cleaned shuffle 371
16/09/07 19:11:31 INFO MemoryStore: ensureFreeSpace(18948072) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:31 INFO MemoryStore: Block broadcast_1139 stored as values in memory (estimated size 18.1 MB, free 3.7 GB)
16/09/07 19:11:31 INFO MemoryStore: ensureFreeSpace(167948) called with curMem=19491896, maxMem=4042889625
16/09/07 19:11:31 INFO MemoryStore: Block broadcast_1139_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:31 INFO BlockManagerInfo: Added broadcast_1139_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:31 INFO SparkContext: Created broadcast 1139 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:31 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:31 INFO DAGScheduler: Registering RDD 2268 (repartition at SparkUtils.java:190)
16/09/07 19:11:31 INFO DAGScheduler: Got job 382 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:31 INFO DAGScheduler: Final stage: ResultStage 755(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:31 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 754)
16/09/07 19:11:31 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 754)
16/09/07 19:11:31 INFO DAGScheduler: Submitting ShuffleMapStage 754 (MapPartitionsRDD[2268] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:31 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19659844, maxMem=4042889625
16/09/07 19:11:31 INFO MemoryStore: Block broadcast_1140 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:31 INFO MemoryStore: ensureFreeSpace(2562) called with curMem=19664308, maxMem=4042889625
16/09/07 19:11:31 INFO MemoryStore: Block broadcast_1140_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:31 INFO BlockManagerInfo: Added broadcast_1140_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:31 INFO SparkContext: Created broadcast 1140 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:31 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 754 (MapPartitionsRDD[2268] at repartition at SparkUtils.java:190)
16/09/07 19:11:31 INFO TaskSchedulerImpl: Adding task set 754.0 with 8 tasks
16/09/07 19:11:31 INFO TaskSetManager: Starting task 0.0 in stage 754.0 (TID 6032, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 1.0 in stage 754.0 (TID 6033, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 2.0 in stage 754.0 (TID 6034, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 3.0 in stage 754.0 (TID 6035, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 4.0 in stage 754.0 (TID 6036, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 5.0 in stage 754.0 (TID 6037, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 6.0 in stage 754.0 (TID 6038, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 7.0 in stage 754.0 (TID 6039, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:31 INFO Executor: Running task 2.0 in stage 754.0 (TID 6034)
16/09/07 19:11:31 INFO Executor: Running task 0.0 in stage 754.0 (TID 6032)
16/09/07 19:11:31 INFO Executor: Running task 5.0 in stage 754.0 (TID 6037)
16/09/07 19:11:31 INFO Executor: Running task 3.0 in stage 754.0 (TID 6035)
16/09/07 19:11:31 INFO Executor: Running task 6.0 in stage 754.0 (TID 6038)
16/09/07 19:11:31 INFO Executor: Running task 1.0 in stage 754.0 (TID 6033)
16/09/07 19:11:31 INFO Executor: Running task 4.0 in stage 754.0 (TID 6036)
16/09/07 19:11:31 INFO Executor: Running task 7.0 in stage 754.0 (TID 6039)
16/09/07 19:11:31 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:31 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:31 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:31 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:31 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:31 INFO Executor: Finished task 5.0 in stage 754.0 (TID 6037). 2260 bytes result sent to driver
16/09/07 19:11:31 INFO Executor: Finished task 7.0 in stage 754.0 (TID 6039). 2260 bytes result sent to driver
16/09/07 19:11:31 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:31 INFO TaskSetManager: Finished task 5.0 in stage 754.0 (TID 6037) in 3 ms on localhost (1/8)
16/09/07 19:11:31 INFO Executor: Finished task 2.0 in stage 754.0 (TID 6034). 2260 bytes result sent to driver
16/09/07 19:11:31 INFO TaskSetManager: Finished task 7.0 in stage 754.0 (TID 6039) in 2 ms on localhost (2/8)
16/09/07 19:11:31 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:31 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:31 INFO TaskSetManager: Finished task 2.0 in stage 754.0 (TID 6034) in 3 ms on localhost (3/8)
16/09/07 19:11:31 INFO Executor: Finished task 4.0 in stage 754.0 (TID 6036). 2260 bytes result sent to driver
16/09/07 19:11:31 INFO Executor: Finished task 3.0 in stage 754.0 (TID 6035). 2260 bytes result sent to driver
16/09/07 19:11:31 INFO Executor: Finished task 6.0 in stage 754.0 (TID 6038). 2260 bytes result sent to driver
16/09/07 19:11:31 INFO Executor: Finished task 1.0 in stage 754.0 (TID 6033). 2260 bytes result sent to driver
16/09/07 19:11:31 INFO TaskSetManager: Finished task 4.0 in stage 754.0 (TID 6036) in 4 ms on localhost (4/8)
16/09/07 19:11:31 INFO Executor: Finished task 0.0 in stage 754.0 (TID 6032). 2260 bytes result sent to driver
16/09/07 19:11:31 INFO TaskSetManager: Finished task 3.0 in stage 754.0 (TID 6035) in 4 ms on localhost (5/8)
16/09/07 19:11:31 INFO TaskSetManager: Finished task 6.0 in stage 754.0 (TID 6038) in 3 ms on localhost (6/8)
16/09/07 19:11:31 INFO TaskSetManager: Finished task 1.0 in stage 754.0 (TID 6033) in 4 ms on localhost (7/8)
16/09/07 19:11:31 INFO TaskSetManager: Finished task 0.0 in stage 754.0 (TID 6032) in 4 ms on localhost (8/8)
16/09/07 19:11:31 INFO TaskSchedulerImpl: Removed TaskSet 754.0, whose tasks have all completed, from pool 
16/09/07 19:11:31 INFO DAGScheduler: ShuffleMapStage 754 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:31 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:31 INFO DAGScheduler: running: Set()
16/09/07 19:11:31 INFO DAGScheduler: waiting: Set(ResultStage 755)
16/09/07 19:11:31 INFO DAGScheduler: failed: Set()
16/09/07 19:11:31 INFO DAGScheduler: Missing parents for ResultStage 755: List()
16/09/07 19:11:31 INFO DAGScheduler: Submitting ResultStage 755 (MapPartitionsRDD[2272] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:31 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19666870, maxMem=4042889625
16/09/07 19:11:31 INFO MemoryStore: Block broadcast_1141 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:31 INFO MemoryStore: ensureFreeSpace(2742) called with curMem=19671966, maxMem=4042889625
16/09/07 19:11:31 INFO MemoryStore: Block broadcast_1141_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:31 INFO BlockManagerInfo: Added broadcast_1141_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:31 INFO SparkContext: Created broadcast 1141 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:31 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 755 (MapPartitionsRDD[2272] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:31 INFO TaskSchedulerImpl: Adding task set 755.0 with 8 tasks
16/09/07 19:11:31 INFO TaskSetManager: Starting task 0.0 in stage 755.0 (TID 6040, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 1.0 in stage 755.0 (TID 6041, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 2.0 in stage 755.0 (TID 6042, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 3.0 in stage 755.0 (TID 6043, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 4.0 in stage 755.0 (TID 6044, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 5.0 in stage 755.0 (TID 6045, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 6.0 in stage 755.0 (TID 6046, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:31 INFO TaskSetManager: Starting task 7.0 in stage 755.0 (TID 6047, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:31 INFO Executor: Running task 1.0 in stage 755.0 (TID 6041)
16/09/07 19:11:31 INFO Executor: Running task 4.0 in stage 755.0 (TID 6044)
16/09/07 19:11:31 INFO Executor: Running task 2.0 in stage 755.0 (TID 6042)
16/09/07 19:11:31 INFO Executor: Running task 0.0 in stage 755.0 (TID 6040)
16/09/07 19:11:31 INFO Executor: Running task 5.0 in stage 755.0 (TID 6045)
16/09/07 19:11:31 INFO Executor: Running task 3.0 in stage 755.0 (TID 6043)
16/09/07 19:11:31 INFO Executor: Running task 7.0 in stage 755.0 (TID 6047)
16/09/07 19:11:31 INFO Executor: Running task 6.0 in stage 755.0 (TID 6046)
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:31 INFO Executor: Finished task 5.0 in stage 755.0 (TID 6045). 2160 bytes result sent to driver
16/09/07 19:11:31 INFO Executor: Finished task 3.0 in stage 755.0 (TID 6043). 2160 bytes result sent to driver
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:31 INFO TaskSetManager: Finished task 5.0 in stage 755.0 (TID 6045) in 2 ms on localhost (1/8)
16/09/07 19:11:31 INFO TaskSetManager: Finished task 3.0 in stage 755.0 (TID 6043) in 2 ms on localhost (2/8)
16/09/07 19:11:31 INFO Executor: Finished task 2.0 in stage 755.0 (TID 6042). 2160 bytes result sent to driver
16/09/07 19:11:31 INFO TaskSetManager: Finished task 2.0 in stage 755.0 (TID 6042) in 2 ms on localhost (3/8)
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:31 INFO Executor: Finished task 4.0 in stage 755.0 (TID 6044). 2160 bytes result sent to driver
16/09/07 19:11:31 INFO TaskSetManager: Finished task 4.0 in stage 755.0 (TID 6044) in 3 ms on localhost (4/8)
16/09/07 19:11:31 INFO BlockManagerInfo: Removed broadcast_1140_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:32 INFO Executor: Finished task 1.0 in stage 755.0 (TID 6041). 3485274 bytes result sent to driver
16/09/07 19:11:32 INFO TaskSetManager: Finished task 1.0 in stage 755.0 (TID 6041) in 588 ms on localhost (5/8)
16/09/07 19:11:32 INFO Executor: Finished task 7.0 in stage 755.0 (TID 6047). 3485524 bytes result sent to driver
16/09/07 19:11:32 INFO Executor: Finished task 6.0 in stage 755.0 (TID 6046). 3485524 bytes result sent to driver
16/09/07 19:11:32 INFO Executor: Finished task 0.0 in stage 755.0 (TID 6040). 3485524 bytes result sent to driver
16/09/07 19:11:32 INFO TaskSetManager: Finished task 0.0 in stage 755.0 (TID 6040) in 919 ms on localhost (6/8)
16/09/07 19:11:32 INFO TaskSetManager: Finished task 7.0 in stage 755.0 (TID 6047) in 930 ms on localhost (7/8)
16/09/07 19:11:32 INFO TaskSetManager: Finished task 6.0 in stage 755.0 (TID 6046) in 932 ms on localhost (8/8)
16/09/07 19:11:32 INFO TaskSchedulerImpl: Removed TaskSet 755.0, whose tasks have all completed, from pool 
16/09/07 19:11:32 INFO DAGScheduler: ResultStage 755 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.932 s
16/09/07 19:11:32 INFO DAGScheduler: Job 382 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.940170 s
16/09/07 19:11:32 INFO ParameterAveragingTrainingMaster: Completed training of split 61 of 78
16/09/07 19:11:32 INFO ParameterAveragingTrainingMaster: Starting training of split 62 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:32 INFO BlockManagerInfo: Removed broadcast_1141_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:32 INFO ContextCleaner: Cleaned accumulator 756
16/09/07 19:11:32 INFO ContextCleaner: Cleaned accumulator 755
16/09/07 19:11:32 INFO BlockManagerInfo: Removed broadcast_1139_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:32 INFO ContextCleaner: Cleaned shuffle 372
16/09/07 19:11:32 INFO MemoryStore: ensureFreeSpace(19383488) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:32 INFO MemoryStore: Block broadcast_1142 stored as values in memory (estimated size 18.5 MB, free 3.7 GB)
16/09/07 19:11:32 INFO MemoryStore: ensureFreeSpace(167953) called with curMem=19927312, maxMem=4042889625
16/09/07 19:11:32 INFO MemoryStore: Block broadcast_1142_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:32 INFO BlockManagerInfo: Added broadcast_1142_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:32 INFO SparkContext: Created broadcast 1142 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:32 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:32 INFO DAGScheduler: Registering RDD 2273 (repartition at SparkUtils.java:190)
16/09/07 19:11:32 INFO DAGScheduler: Got job 383 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:32 INFO DAGScheduler: Final stage: ResultStage 757(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:32 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 756)
16/09/07 19:11:32 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 756)
16/09/07 19:11:32 INFO DAGScheduler: Submitting ShuffleMapStage 756 (MapPartitionsRDD[2273] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:32 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=20095265, maxMem=4042889625
16/09/07 19:11:32 INFO MemoryStore: Block broadcast_1143 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:32 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=20099729, maxMem=4042889625
16/09/07 19:11:32 INFO MemoryStore: Block broadcast_1143_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:32 INFO BlockManagerInfo: Added broadcast_1143_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:32 INFO SparkContext: Created broadcast 1143 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:32 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 756 (MapPartitionsRDD[2273] at repartition at SparkUtils.java:190)
16/09/07 19:11:32 INFO TaskSchedulerImpl: Adding task set 756.0 with 8 tasks
16/09/07 19:11:32 INFO TaskSetManager: Starting task 0.0 in stage 756.0 (TID 6048, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 1.0 in stage 756.0 (TID 6049, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 2.0 in stage 756.0 (TID 6050, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 3.0 in stage 756.0 (TID 6051, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 4.0 in stage 756.0 (TID 6052, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 5.0 in stage 756.0 (TID 6053, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 6.0 in stage 756.0 (TID 6054, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 7.0 in stage 756.0 (TID 6055, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:32 INFO Executor: Running task 1.0 in stage 756.0 (TID 6049)
16/09/07 19:11:32 INFO Executor: Running task 3.0 in stage 756.0 (TID 6051)
16/09/07 19:11:32 INFO Executor: Running task 4.0 in stage 756.0 (TID 6052)
16/09/07 19:11:32 INFO Executor: Running task 5.0 in stage 756.0 (TID 6053)
16/09/07 19:11:32 INFO Executor: Running task 2.0 in stage 756.0 (TID 6050)
16/09/07 19:11:32 INFO Executor: Running task 6.0 in stage 756.0 (TID 6054)
16/09/07 19:11:32 INFO Executor: Running task 0.0 in stage 756.0 (TID 6048)
16/09/07 19:11:32 INFO Executor: Running task 7.0 in stage 756.0 (TID 6055)
16/09/07 19:11:32 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:32 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:32 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:32 INFO Executor: Finished task 4.0 in stage 756.0 (TID 6052). 2260 bytes result sent to driver
16/09/07 19:11:32 INFO TaskSetManager: Finished task 4.0 in stage 756.0 (TID 6052) in 2 ms on localhost (1/8)
16/09/07 19:11:32 INFO Executor: Finished task 5.0 in stage 756.0 (TID 6053). 2260 bytes result sent to driver
16/09/07 19:11:32 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:32 INFO Executor: Finished task 7.0 in stage 756.0 (TID 6055). 2260 bytes result sent to driver
16/09/07 19:11:32 INFO TaskSetManager: Finished task 5.0 in stage 756.0 (TID 6053) in 2 ms on localhost (2/8)
16/09/07 19:11:32 INFO TaskSetManager: Finished task 7.0 in stage 756.0 (TID 6055) in 2 ms on localhost (3/8)
16/09/07 19:11:32 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:32 INFO Executor: Finished task 1.0 in stage 756.0 (TID 6049). 2260 bytes result sent to driver
16/09/07 19:11:32 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:32 INFO TaskSetManager: Finished task 1.0 in stage 756.0 (TID 6049) in 3 ms on localhost (4/8)
16/09/07 19:11:32 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:32 INFO Executor: Finished task 2.0 in stage 756.0 (TID 6050). 2260 bytes result sent to driver
16/09/07 19:11:32 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:32 INFO TaskSetManager: Finished task 2.0 in stage 756.0 (TID 6050) in 4 ms on localhost (5/8)
16/09/07 19:11:32 INFO Executor: Finished task 0.0 in stage 756.0 (TID 6048). 2260 bytes result sent to driver
16/09/07 19:11:32 INFO TaskSetManager: Finished task 0.0 in stage 756.0 (TID 6048) in 4 ms on localhost (6/8)
16/09/07 19:11:32 INFO Executor: Finished task 6.0 in stage 756.0 (TID 6054). 2260 bytes result sent to driver
16/09/07 19:11:32 INFO Executor: Finished task 3.0 in stage 756.0 (TID 6051). 2260 bytes result sent to driver
16/09/07 19:11:32 INFO TaskSetManager: Finished task 6.0 in stage 756.0 (TID 6054) in 4 ms on localhost (7/8)
16/09/07 19:11:32 INFO TaskSetManager: Finished task 3.0 in stage 756.0 (TID 6051) in 5 ms on localhost (8/8)
16/09/07 19:11:32 INFO DAGScheduler: ShuffleMapStage 756 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:11:32 INFO TaskSchedulerImpl: Removed TaskSet 756.0, whose tasks have all completed, from pool 
16/09/07 19:11:32 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:32 INFO DAGScheduler: running: Set()
16/09/07 19:11:32 INFO DAGScheduler: waiting: Set(ResultStage 757)
16/09/07 19:11:32 INFO DAGScheduler: failed: Set()
16/09/07 19:11:32 INFO DAGScheduler: Missing parents for ResultStage 757: List()
16/09/07 19:11:32 INFO DAGScheduler: Submitting ResultStage 757 (MapPartitionsRDD[2277] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:32 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=20102293, maxMem=4042889625
16/09/07 19:11:32 INFO MemoryStore: Block broadcast_1144 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:32 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=20107389, maxMem=4042889625
16/09/07 19:11:32 INFO MemoryStore: Block broadcast_1144_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:32 INFO BlockManagerInfo: Added broadcast_1144_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:32 INFO SparkContext: Created broadcast 1144 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:32 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 757 (MapPartitionsRDD[2277] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:32 INFO TaskSchedulerImpl: Adding task set 757.0 with 8 tasks
16/09/07 19:11:32 INFO TaskSetManager: Starting task 0.0 in stage 757.0 (TID 6056, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 1.0 in stage 757.0 (TID 6057, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 2.0 in stage 757.0 (TID 6058, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 3.0 in stage 757.0 (TID 6059, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 4.0 in stage 757.0 (TID 6060, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 5.0 in stage 757.0 (TID 6061, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 6.0 in stage 757.0 (TID 6062, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:32 INFO TaskSetManager: Starting task 7.0 in stage 757.0 (TID 6063, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:32 INFO Executor: Running task 0.0 in stage 757.0 (TID 6056)
16/09/07 19:11:32 INFO Executor: Running task 4.0 in stage 757.0 (TID 6060)
16/09/07 19:11:32 INFO Executor: Running task 7.0 in stage 757.0 (TID 6063)
16/09/07 19:11:32 INFO Executor: Running task 3.0 in stage 757.0 (TID 6059)
16/09/07 19:11:32 INFO Executor: Running task 6.0 in stage 757.0 (TID 6062)
16/09/07 19:11:32 INFO Executor: Running task 2.0 in stage 757.0 (TID 6058)
16/09/07 19:11:32 INFO Executor: Running task 1.0 in stage 757.0 (TID 6057)
16/09/07 19:11:32 INFO Executor: Running task 5.0 in stage 757.0 (TID 6061)
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:32 INFO Executor: Finished task 5.0 in stage 757.0 (TID 6061). 2160 bytes result sent to driver
16/09/07 19:11:32 INFO Executor: Finished task 2.0 in stage 757.0 (TID 6058). 2160 bytes result sent to driver
16/09/07 19:11:32 INFO Executor: Finished task 1.0 in stage 757.0 (TID 6057). 2160 bytes result sent to driver
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:32 INFO TaskSetManager: Finished task 5.0 in stage 757.0 (TID 6061) in 2 ms on localhost (1/8)
16/09/07 19:11:32 INFO TaskSetManager: Finished task 1.0 in stage 757.0 (TID 6057) in 2 ms on localhost (2/8)
16/09/07 19:11:32 INFO TaskSetManager: Finished task 2.0 in stage 757.0 (TID 6058) in 2 ms on localhost (3/8)
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:32 INFO Executor: Finished task 4.0 in stage 757.0 (TID 6060). 2160 bytes result sent to driver
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:32 INFO TaskSetManager: Finished task 4.0 in stage 757.0 (TID 6060) in 3 ms on localhost (4/8)
16/09/07 19:11:32 INFO Executor: Finished task 3.0 in stage 757.0 (TID 6059). 2160 bytes result sent to driver
16/09/07 19:11:32 INFO TaskSetManager: Finished task 3.0 in stage 757.0 (TID 6059) in 3 ms on localhost (5/8)
16/09/07 19:11:33 INFO BlockManagerInfo: Removed broadcast_1143_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:33 INFO Executor: Finished task 0.0 in stage 757.0 (TID 6056). 3485274 bytes result sent to driver
16/09/07 19:11:33 INFO Executor: Finished task 7.0 in stage 757.0 (TID 6063). 3485274 bytes result sent to driver
16/09/07 19:11:33 INFO Executor: Finished task 6.0 in stage 757.0 (TID 6062). 3485274 bytes result sent to driver
16/09/07 19:11:33 INFO TaskSetManager: Finished task 6.0 in stage 757.0 (TID 6062) in 934 ms on localhost (6/8)
16/09/07 19:11:33 INFO TaskSetManager: Finished task 7.0 in stage 757.0 (TID 6063) in 935 ms on localhost (7/8)
16/09/07 19:11:33 INFO TaskSetManager: Finished task 0.0 in stage 757.0 (TID 6056) in 936 ms on localhost (8/8)
16/09/07 19:11:33 INFO TaskSchedulerImpl: Removed TaskSet 757.0, whose tasks have all completed, from pool 
16/09/07 19:11:33 INFO DAGScheduler: ResultStage 757 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.936 s
16/09/07 19:11:33 INFO DAGScheduler: Job 383 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.944194 s
16/09/07 19:11:33 INFO ParameterAveragingTrainingMaster: Completed training of split 62 of 78
16/09/07 19:11:33 INFO ParameterAveragingTrainingMaster: Starting training of split 63 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:34 INFO MemoryStore: ensureFreeSpace(19199392) called with curMem=20103104, maxMem=4042889625
16/09/07 19:11:34 INFO MemoryStore: Block broadcast_1145 stored as values in memory (estimated size 18.3 MB, free 3.7 GB)
16/09/07 19:11:34 INFO MemoryStore: ensureFreeSpace(167941) called with curMem=39302496, maxMem=4042889625
16/09/07 19:11:34 INFO MemoryStore: Block broadcast_1145_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:34 INFO BlockManagerInfo: Added broadcast_1145_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:34 INFO SparkContext: Created broadcast 1145 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:34 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:34 INFO DAGScheduler: Registering RDD 2278 (repartition at SparkUtils.java:190)
16/09/07 19:11:34 INFO DAGScheduler: Got job 384 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:34 INFO DAGScheduler: Final stage: ResultStage 759(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:34 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 758)
16/09/07 19:11:34 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 758)
16/09/07 19:11:34 INFO DAGScheduler: Submitting ShuffleMapStage 758 (MapPartitionsRDD[2278] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:34 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=39470437, maxMem=4042889625
16/09/07 19:11:34 INFO MemoryStore: Block broadcast_1146 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:34 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=39474901, maxMem=4042889625
16/09/07 19:11:34 INFO MemoryStore: Block broadcast_1146_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:34 INFO BlockManagerInfo: Added broadcast_1146_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:34 INFO SparkContext: Created broadcast 1146 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:34 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 758 (MapPartitionsRDD[2278] at repartition at SparkUtils.java:190)
16/09/07 19:11:34 INFO TaskSchedulerImpl: Adding task set 758.0 with 8 tasks
16/09/07 19:11:34 INFO TaskSetManager: Starting task 0.0 in stage 758.0 (TID 6064, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 1.0 in stage 758.0 (TID 6065, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 2.0 in stage 758.0 (TID 6066, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 3.0 in stage 758.0 (TID 6067, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 4.0 in stage 758.0 (TID 6068, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 5.0 in stage 758.0 (TID 6069, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 6.0 in stage 758.0 (TID 6070, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 7.0 in stage 758.0 (TID 6071, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:34 INFO Executor: Running task 0.0 in stage 758.0 (TID 6064)
16/09/07 19:11:34 INFO Executor: Running task 4.0 in stage 758.0 (TID 6068)
16/09/07 19:11:34 INFO Executor: Running task 7.0 in stage 758.0 (TID 6071)
16/09/07 19:11:34 INFO Executor: Running task 6.0 in stage 758.0 (TID 6070)
16/09/07 19:11:34 INFO Executor: Running task 1.0 in stage 758.0 (TID 6065)
16/09/07 19:11:34 INFO Executor: Running task 3.0 in stage 758.0 (TID 6067)
16/09/07 19:11:34 INFO Executor: Running task 5.0 in stage 758.0 (TID 6069)
16/09/07 19:11:34 INFO Executor: Running task 2.0 in stage 758.0 (TID 6066)
16/09/07 19:11:34 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:34 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:34 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:34 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:34 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:34 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:34 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:34 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:34 INFO Executor: Finished task 4.0 in stage 758.0 (TID 6068). 2260 bytes result sent to driver
16/09/07 19:11:34 INFO Executor: Finished task 1.0 in stage 758.0 (TID 6065). 2260 bytes result sent to driver
16/09/07 19:11:34 INFO Executor: Finished task 6.0 in stage 758.0 (TID 6070). 2260 bytes result sent to driver
16/09/07 19:11:34 INFO Executor: Finished task 7.0 in stage 758.0 (TID 6071). 2260 bytes result sent to driver
16/09/07 19:11:34 INFO TaskSetManager: Finished task 4.0 in stage 758.0 (TID 6068) in 2 ms on localhost (1/8)
16/09/07 19:11:34 INFO TaskSetManager: Finished task 1.0 in stage 758.0 (TID 6065) in 3 ms on localhost (2/8)
16/09/07 19:11:34 INFO TaskSetManager: Finished task 6.0 in stage 758.0 (TID 6070) in 3 ms on localhost (3/8)
16/09/07 19:11:34 INFO TaskSetManager: Finished task 7.0 in stage 758.0 (TID 6071) in 3 ms on localhost (4/8)
16/09/07 19:11:34 INFO Executor: Finished task 0.0 in stage 758.0 (TID 6064). 2260 bytes result sent to driver
16/09/07 19:11:34 INFO Executor: Finished task 5.0 in stage 758.0 (TID 6069). 2260 bytes result sent to driver
16/09/07 19:11:34 INFO Executor: Finished task 2.0 in stage 758.0 (TID 6066). 2260 bytes result sent to driver
16/09/07 19:11:34 INFO TaskSetManager: Finished task 0.0 in stage 758.0 (TID 6064) in 3 ms on localhost (5/8)
16/09/07 19:11:34 INFO Executor: Finished task 3.0 in stage 758.0 (TID 6067). 2260 bytes result sent to driver
16/09/07 19:11:34 INFO TaskSetManager: Finished task 5.0 in stage 758.0 (TID 6069) in 3 ms on localhost (6/8)
16/09/07 19:11:34 INFO TaskSetManager: Finished task 2.0 in stage 758.0 (TID 6066) in 3 ms on localhost (7/8)
16/09/07 19:11:34 INFO TaskSetManager: Finished task 3.0 in stage 758.0 (TID 6067) in 3 ms on localhost (8/8)
16/09/07 19:11:34 INFO TaskSchedulerImpl: Removed TaskSet 758.0, whose tasks have all completed, from pool 
16/09/07 19:11:34 INFO DAGScheduler: ShuffleMapStage 758 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:34 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:34 INFO DAGScheduler: running: Set()
16/09/07 19:11:34 INFO DAGScheduler: waiting: Set(ResultStage 759)
16/09/07 19:11:34 INFO DAGScheduler: failed: Set()
16/09/07 19:11:34 INFO DAGScheduler: Missing parents for ResultStage 759: List()
16/09/07 19:11:34 INFO DAGScheduler: Submitting ResultStage 759 (MapPartitionsRDD[2282] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:34 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=39477464, maxMem=4042889625
16/09/07 19:11:34 INFO MemoryStore: Block broadcast_1147 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:34 INFO MemoryStore: ensureFreeSpace(2742) called with curMem=39482560, maxMem=4042889625
16/09/07 19:11:34 INFO MemoryStore: Block broadcast_1147_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:34 INFO BlockManagerInfo: Added broadcast_1147_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:34 INFO SparkContext: Created broadcast 1147 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:34 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 759 (MapPartitionsRDD[2282] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:34 INFO TaskSchedulerImpl: Adding task set 759.0 with 8 tasks
16/09/07 19:11:34 INFO TaskSetManager: Starting task 0.0 in stage 759.0 (TID 6072, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 1.0 in stage 759.0 (TID 6073, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 2.0 in stage 759.0 (TID 6074, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 3.0 in stage 759.0 (TID 6075, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 4.0 in stage 759.0 (TID 6076, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 5.0 in stage 759.0 (TID 6077, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 6.0 in stage 759.0 (TID 6078, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:34 INFO TaskSetManager: Starting task 7.0 in stage 759.0 (TID 6079, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:34 INFO Executor: Running task 1.0 in stage 759.0 (TID 6073)
16/09/07 19:11:34 INFO Executor: Running task 3.0 in stage 759.0 (TID 6075)
16/09/07 19:11:34 INFO Executor: Running task 7.0 in stage 759.0 (TID 6079)
16/09/07 19:11:34 INFO Executor: Running task 0.0 in stage 759.0 (TID 6072)
16/09/07 19:11:34 INFO Executor: Running task 2.0 in stage 759.0 (TID 6074)
16/09/07 19:11:34 INFO Executor: Running task 6.0 in stage 759.0 (TID 6078)
16/09/07 19:11:34 INFO Executor: Running task 5.0 in stage 759.0 (TID 6077)
16/09/07 19:11:34 INFO Executor: Running task 4.0 in stage 759.0 (TID 6076)
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:34 INFO Executor: Finished task 4.0 in stage 759.0 (TID 6076). 2160 bytes result sent to driver
16/09/07 19:11:34 INFO Executor: Finished task 5.0 in stage 759.0 (TID 6077). 2160 bytes result sent to driver
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:34 INFO TaskSetManager: Finished task 5.0 in stage 759.0 (TID 6077) in 1 ms on localhost (1/8)
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:34 INFO Executor: Finished task 3.0 in stage 759.0 (TID 6075). 2160 bytes result sent to driver
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:34 INFO TaskSetManager: Finished task 4.0 in stage 759.0 (TID 6076) in 1 ms on localhost (2/8)
16/09/07 19:11:34 INFO TaskSetManager: Finished task 3.0 in stage 759.0 (TID 6075) in 2 ms on localhost (3/8)
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:34 INFO Executor: Finished task 2.0 in stage 759.0 (TID 6074). 2160 bytes result sent to driver
16/09/07 19:11:34 INFO TaskSetManager: Finished task 2.0 in stage 759.0 (TID 6074) in 2 ms on localhost (4/8)
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:34 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:34 INFO Executor: Finished task 1.0 in stage 759.0 (TID 6073). 2160 bytes result sent to driver
16/09/07 19:11:34 INFO TaskSetManager: Finished task 1.0 in stage 759.0 (TID 6073) in 4 ms on localhost (5/8)
16/09/07 19:11:34 INFO BlockManagerInfo: Removed broadcast_1144_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:34 INFO BlockManagerInfo: Removed broadcast_1146_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:34 INFO ContextCleaner: Cleaned accumulator 758
16/09/07 19:11:34 INFO ContextCleaner: Cleaned accumulator 757
16/09/07 19:11:34 INFO BlockManagerInfo: Removed broadcast_1142_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:34 INFO ContextCleaner: Cleaned shuffle 373
16/09/07 19:11:34 INFO Executor: Finished task 0.0 in stage 759.0 (TID 6072). 3485274 bytes result sent to driver
16/09/07 19:11:34 INFO TaskSetManager: Finished task 0.0 in stage 759.0 (TID 6072) in 870 ms on localhost (6/8)
16/09/07 19:11:35 INFO Executor: Finished task 6.0 in stage 759.0 (TID 6078). 3485274 bytes result sent to driver
16/09/07 19:11:35 INFO Executor: Finished task 7.0 in stage 759.0 (TID 6079). 3485274 bytes result sent to driver
16/09/07 19:11:35 INFO TaskSetManager: Finished task 6.0 in stage 759.0 (TID 6078) in 931 ms on localhost (7/8)
16/09/07 19:11:35 INFO TaskSetManager: Finished task 7.0 in stage 759.0 (TID 6079) in 933 ms on localhost (8/8)
16/09/07 19:11:35 INFO TaskSchedulerImpl: Removed TaskSet 759.0, whose tasks have all completed, from pool 
16/09/07 19:11:35 INFO DAGScheduler: ResultStage 759 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.934 s
16/09/07 19:11:35 INFO DAGScheduler: Job 384 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.941412 s
16/09/07 19:11:35 INFO ParameterAveragingTrainingMaster: Completed training of split 63 of 78
16/09/07 19:11:35 INFO ParameterAveragingTrainingMaster: Starting training of split 64 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:35 INFO BlockManagerInfo: Removed broadcast_1147_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:35 INFO ContextCleaner: Cleaned accumulator 760
16/09/07 19:11:35 INFO ContextCleaner: Cleaned accumulator 759
16/09/07 19:11:35 INFO BlockManagerInfo: Removed broadcast_1145_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:35 INFO ContextCleaner: Cleaned shuffle 374
16/09/07 19:11:35 INFO MemoryStore: ensureFreeSpace(19035392) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:35 INFO MemoryStore: Block broadcast_1148 stored as values in memory (estimated size 18.2 MB, free 3.7 GB)
16/09/07 19:11:35 INFO MemoryStore: ensureFreeSpace(167931) called with curMem=19579216, maxMem=4042889625
16/09/07 19:11:35 INFO MemoryStore: Block broadcast_1148_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:35 INFO BlockManagerInfo: Added broadcast_1148_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:35 INFO SparkContext: Created broadcast 1148 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:35 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:35 INFO DAGScheduler: Registering RDD 2283 (repartition at SparkUtils.java:190)
16/09/07 19:11:35 INFO DAGScheduler: Got job 385 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:35 INFO DAGScheduler: Final stage: ResultStage 761(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:35 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 760)
16/09/07 19:11:35 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 760)
16/09/07 19:11:35 INFO DAGScheduler: Submitting ShuffleMapStage 760 (MapPartitionsRDD[2283] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:35 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19747147, maxMem=4042889625
16/09/07 19:11:35 INFO MemoryStore: Block broadcast_1149 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:35 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19751611, maxMem=4042889625
16/09/07 19:11:35 INFO MemoryStore: Block broadcast_1149_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:35 INFO BlockManagerInfo: Added broadcast_1149_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:35 INFO SparkContext: Created broadcast 1149 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:35 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 760 (MapPartitionsRDD[2283] at repartition at SparkUtils.java:190)
16/09/07 19:11:35 INFO TaskSchedulerImpl: Adding task set 760.0 with 8 tasks
16/09/07 19:11:35 INFO TaskSetManager: Starting task 0.0 in stage 760.0 (TID 6080, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 1.0 in stage 760.0 (TID 6081, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 2.0 in stage 760.0 (TID 6082, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 3.0 in stage 760.0 (TID 6083, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 4.0 in stage 760.0 (TID 6084, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 5.0 in stage 760.0 (TID 6085, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 6.0 in stage 760.0 (TID 6086, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 7.0 in stage 760.0 (TID 6087, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:35 INFO Executor: Running task 3.0 in stage 760.0 (TID 6083)
16/09/07 19:11:35 INFO Executor: Running task 2.0 in stage 760.0 (TID 6082)
16/09/07 19:11:35 INFO Executor: Running task 4.0 in stage 760.0 (TID 6084)
16/09/07 19:11:35 INFO Executor: Running task 6.0 in stage 760.0 (TID 6086)
16/09/07 19:11:35 INFO Executor: Running task 5.0 in stage 760.0 (TID 6085)
16/09/07 19:11:35 INFO Executor: Running task 0.0 in stage 760.0 (TID 6080)
16/09/07 19:11:35 INFO Executor: Running task 1.0 in stage 760.0 (TID 6081)
16/09/07 19:11:35 INFO Executor: Running task 7.0 in stage 760.0 (TID 6087)
16/09/07 19:11:35 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:35 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:35 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:35 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:35 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:35 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:35 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:35 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:35 INFO Executor: Finished task 0.0 in stage 760.0 (TID 6080). 2260 bytes result sent to driver
16/09/07 19:11:35 INFO Executor: Finished task 3.0 in stage 760.0 (TID 6083). 2260 bytes result sent to driver
16/09/07 19:11:35 INFO Executor: Finished task 7.0 in stage 760.0 (TID 6087). 2260 bytes result sent to driver
16/09/07 19:11:35 INFO Executor: Finished task 5.0 in stage 760.0 (TID 6085). 2260 bytes result sent to driver
16/09/07 19:11:35 INFO TaskSetManager: Finished task 3.0 in stage 760.0 (TID 6083) in 3 ms on localhost (1/8)
16/09/07 19:11:35 INFO TaskSetManager: Finished task 7.0 in stage 760.0 (TID 6087) in 2 ms on localhost (2/8)
16/09/07 19:11:35 INFO TaskSetManager: Finished task 0.0 in stage 760.0 (TID 6080) in 3 ms on localhost (3/8)
16/09/07 19:11:35 INFO TaskSetManager: Finished task 5.0 in stage 760.0 (TID 6085) in 3 ms on localhost (4/8)
16/09/07 19:11:35 INFO Executor: Finished task 4.0 in stage 760.0 (TID 6084). 2260 bytes result sent to driver
16/09/07 19:11:35 INFO TaskSetManager: Finished task 4.0 in stage 760.0 (TID 6084) in 4 ms on localhost (5/8)
16/09/07 19:11:35 INFO Executor: Finished task 1.0 in stage 760.0 (TID 6081). 2260 bytes result sent to driver
16/09/07 19:11:35 INFO Executor: Finished task 2.0 in stage 760.0 (TID 6082). 2260 bytes result sent to driver
16/09/07 19:11:35 INFO TaskSetManager: Finished task 1.0 in stage 760.0 (TID 6081) in 4 ms on localhost (6/8)
16/09/07 19:11:35 INFO Executor: Finished task 6.0 in stage 760.0 (TID 6086). 2260 bytes result sent to driver
16/09/07 19:11:35 INFO TaskSetManager: Finished task 2.0 in stage 760.0 (TID 6082) in 4 ms on localhost (7/8)
16/09/07 19:11:35 INFO TaskSetManager: Finished task 6.0 in stage 760.0 (TID 6086) in 3 ms on localhost (8/8)
16/09/07 19:11:35 INFO TaskSchedulerImpl: Removed TaskSet 760.0, whose tasks have all completed, from pool 
16/09/07 19:11:35 INFO DAGScheduler: ShuffleMapStage 760 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:35 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:35 INFO DAGScheduler: running: Set()
16/09/07 19:11:35 INFO DAGScheduler: waiting: Set(ResultStage 761)
16/09/07 19:11:35 INFO DAGScheduler: failed: Set()
16/09/07 19:11:35 INFO DAGScheduler: Missing parents for ResultStage 761: List()
16/09/07 19:11:35 INFO DAGScheduler: Submitting ResultStage 761 (MapPartitionsRDD[2287] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:35 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19754175, maxMem=4042889625
16/09/07 19:11:35 INFO MemoryStore: Block broadcast_1150 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:35 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=19759271, maxMem=4042889625
16/09/07 19:11:35 INFO MemoryStore: Block broadcast_1150_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:35 INFO BlockManagerInfo: Added broadcast_1150_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:35 INFO SparkContext: Created broadcast 1150 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:35 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 761 (MapPartitionsRDD[2287] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:35 INFO TaskSchedulerImpl: Adding task set 761.0 with 8 tasks
16/09/07 19:11:35 INFO TaskSetManager: Starting task 0.0 in stage 761.0 (TID 6088, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 1.0 in stage 761.0 (TID 6089, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 2.0 in stage 761.0 (TID 6090, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 3.0 in stage 761.0 (TID 6091, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 4.0 in stage 761.0 (TID 6092, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 5.0 in stage 761.0 (TID 6093, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 6.0 in stage 761.0 (TID 6094, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:35 INFO TaskSetManager: Starting task 7.0 in stage 761.0 (TID 6095, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:35 INFO Executor: Running task 1.0 in stage 761.0 (TID 6089)
16/09/07 19:11:35 INFO Executor: Running task 3.0 in stage 761.0 (TID 6091)
16/09/07 19:11:35 INFO Executor: Running task 7.0 in stage 761.0 (TID 6095)
16/09/07 19:11:35 INFO Executor: Running task 4.0 in stage 761.0 (TID 6092)
16/09/07 19:11:35 INFO Executor: Running task 0.0 in stage 761.0 (TID 6088)
16/09/07 19:11:35 INFO Executor: Running task 2.0 in stage 761.0 (TID 6090)
16/09/07 19:11:35 INFO Executor: Running task 5.0 in stage 761.0 (TID 6093)
16/09/07 19:11:35 INFO Executor: Running task 6.0 in stage 761.0 (TID 6094)
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:35 INFO Executor: Finished task 4.0 in stage 761.0 (TID 6092). 2160 bytes result sent to driver
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:35 INFO Executor: Finished task 5.0 in stage 761.0 (TID 6093). 2160 bytes result sent to driver
16/09/07 19:11:35 INFO Executor: Finished task 2.0 in stage 761.0 (TID 6090). 2160 bytes result sent to driver
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:35 INFO TaskSetManager: Finished task 2.0 in stage 761.0 (TID 6090) in 2 ms on localhost (1/8)
16/09/07 19:11:35 INFO TaskSetManager: Finished task 5.0 in stage 761.0 (TID 6093) in 2 ms on localhost (2/8)
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:35 INFO TaskSetManager: Finished task 4.0 in stage 761.0 (TID 6092) in 2 ms on localhost (3/8)
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:35 INFO Executor: Finished task 1.0 in stage 761.0 (TID 6089). 2160 bytes result sent to driver
16/09/07 19:11:35 INFO TaskSetManager: Finished task 1.0 in stage 761.0 (TID 6089) in 3 ms on localhost (4/8)
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:35 INFO Executor: Finished task 3.0 in stage 761.0 (TID 6091). 2160 bytes result sent to driver
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:35 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:35 INFO TaskSetManager: Finished task 3.0 in stage 761.0 (TID 6091) in 3 ms on localhost (5/8)
16/09/07 19:11:35 INFO BlockManagerInfo: Removed broadcast_1149_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:36 INFO Executor: Finished task 7.0 in stage 761.0 (TID 6095). 3485274 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 7.0 in stage 761.0 (TID 6095) in 885 ms on localhost (6/8)
16/09/07 19:11:36 INFO Executor: Finished task 0.0 in stage 761.0 (TID 6088). 3485274 bytes result sent to driver
16/09/07 19:11:36 INFO Executor: Finished task 6.0 in stage 761.0 (TID 6094). 3485274 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 0.0 in stage 761.0 (TID 6088) in 1046 ms on localhost (7/8)
16/09/07 19:11:36 INFO TaskSetManager: Finished task 6.0 in stage 761.0 (TID 6094) in 1046 ms on localhost (8/8)
16/09/07 19:11:36 INFO TaskSchedulerImpl: Removed TaskSet 761.0, whose tasks have all completed, from pool 
16/09/07 19:11:36 INFO DAGScheduler: ResultStage 761 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 1.047 s
16/09/07 19:11:36 INFO DAGScheduler: Job 385 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 1.054780 s
16/09/07 19:11:36 INFO ParameterAveragingTrainingMaster: Completed training of split 64 of 78
16/09/07 19:11:36 INFO ParameterAveragingTrainingMaster: Starting training of split 65 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:36 INFO MemoryStore: ensureFreeSpace(19225984) called with curMem=19754987, maxMem=4042889625
16/09/07 19:11:36 INFO MemoryStore: Block broadcast_1151 stored as values in memory (estimated size 18.3 MB, free 3.7 GB)
16/09/07 19:11:36 INFO MemoryStore: ensureFreeSpace(167996) called with curMem=38980971, maxMem=4042889625
16/09/07 19:11:36 INFO MemoryStore: Block broadcast_1151_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:36 INFO BlockManagerInfo: Added broadcast_1151_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:36 INFO SparkContext: Created broadcast 1151 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:36 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:36 INFO DAGScheduler: Registering RDD 2288 (repartition at SparkUtils.java:190)
16/09/07 19:11:36 INFO DAGScheduler: Got job 386 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:36 INFO DAGScheduler: Final stage: ResultStage 763(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:36 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 762)
16/09/07 19:11:36 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 762)
16/09/07 19:11:36 INFO DAGScheduler: Submitting ShuffleMapStage 762 (MapPartitionsRDD[2288] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:36 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=39148967, maxMem=4042889625
16/09/07 19:11:36 INFO MemoryStore: Block broadcast_1152 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:36 INFO MemoryStore: ensureFreeSpace(2561) called with curMem=39153431, maxMem=4042889625
16/09/07 19:11:36 INFO MemoryStore: Block broadcast_1152_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:36 INFO BlockManagerInfo: Added broadcast_1152_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:36 INFO SparkContext: Created broadcast 1152 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:36 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 762 (MapPartitionsRDD[2288] at repartition at SparkUtils.java:190)
16/09/07 19:11:36 INFO TaskSchedulerImpl: Adding task set 762.0 with 8 tasks
16/09/07 19:11:36 INFO TaskSetManager: Starting task 0.0 in stage 762.0 (TID 6096, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 1.0 in stage 762.0 (TID 6097, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 2.0 in stage 762.0 (TID 6098, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 3.0 in stage 762.0 (TID 6099, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 4.0 in stage 762.0 (TID 6100, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 5.0 in stage 762.0 (TID 6101, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 6.0 in stage 762.0 (TID 6102, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 7.0 in stage 762.0 (TID 6103, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:36 INFO Executor: Running task 1.0 in stage 762.0 (TID 6097)
16/09/07 19:11:36 INFO Executor: Running task 0.0 in stage 762.0 (TID 6096)
16/09/07 19:11:36 INFO Executor: Running task 7.0 in stage 762.0 (TID 6103)
16/09/07 19:11:36 INFO Executor: Running task 6.0 in stage 762.0 (TID 6102)
16/09/07 19:11:36 INFO Executor: Running task 2.0 in stage 762.0 (TID 6098)
16/09/07 19:11:36 INFO Executor: Running task 3.0 in stage 762.0 (TID 6099)
16/09/07 19:11:36 INFO Executor: Running task 5.0 in stage 762.0 (TID 6101)
16/09/07 19:11:36 INFO Executor: Running task 4.0 in stage 762.0 (TID 6100)
16/09/07 19:11:36 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:36 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:36 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:36 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:36 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:36 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:36 INFO Executor: Finished task 6.0 in stage 762.0 (TID 6102). 2260 bytes result sent to driver
16/09/07 19:11:36 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:36 INFO Executor: Finished task 3.0 in stage 762.0 (TID 6099). 2260 bytes result sent to driver
16/09/07 19:11:36 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:36 INFO Executor: Finished task 0.0 in stage 762.0 (TID 6096). 2260 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 3.0 in stage 762.0 (TID 6099) in 2 ms on localhost (1/8)
16/09/07 19:11:36 INFO TaskSetManager: Finished task 6.0 in stage 762.0 (TID 6102) in 2 ms on localhost (2/8)
16/09/07 19:11:36 INFO Executor: Finished task 1.0 in stage 762.0 (TID 6097). 2260 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 0.0 in stage 762.0 (TID 6096) in 2 ms on localhost (3/8)
16/09/07 19:11:36 INFO Executor: Finished task 4.0 in stage 762.0 (TID 6100). 2260 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 1.0 in stage 762.0 (TID 6097) in 3 ms on localhost (4/8)
16/09/07 19:11:36 INFO Executor: Finished task 7.0 in stage 762.0 (TID 6103). 2260 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 4.0 in stage 762.0 (TID 6100) in 3 ms on localhost (5/8)
16/09/07 19:11:36 INFO TaskSetManager: Finished task 7.0 in stage 762.0 (TID 6103) in 3 ms on localhost (6/8)
16/09/07 19:11:36 INFO Executor: Finished task 2.0 in stage 762.0 (TID 6098). 2260 bytes result sent to driver
16/09/07 19:11:36 INFO Executor: Finished task 5.0 in stage 762.0 (TID 6101). 2260 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 2.0 in stage 762.0 (TID 6098) in 3 ms on localhost (7/8)
16/09/07 19:11:36 INFO TaskSetManager: Finished task 5.0 in stage 762.0 (TID 6101) in 3 ms on localhost (8/8)
16/09/07 19:11:36 INFO TaskSchedulerImpl: Removed TaskSet 762.0, whose tasks have all completed, from pool 
16/09/07 19:11:36 INFO DAGScheduler: ShuffleMapStage 762 (repartition at SparkUtils.java:190) finished in 0.003 s
16/09/07 19:11:36 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:36 INFO DAGScheduler: running: Set()
16/09/07 19:11:36 INFO DAGScheduler: waiting: Set(ResultStage 763)
16/09/07 19:11:36 INFO DAGScheduler: failed: Set()
16/09/07 19:11:36 INFO DAGScheduler: Missing parents for ResultStage 763: List()
16/09/07 19:11:36 INFO DAGScheduler: Submitting ResultStage 763 (MapPartitionsRDD[2292] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:36 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=39155992, maxMem=4042889625
16/09/07 19:11:36 INFO MemoryStore: Block broadcast_1153 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:36 INFO MemoryStore: ensureFreeSpace(2740) called with curMem=39161088, maxMem=4042889625
16/09/07 19:11:36 INFO MemoryStore: Block broadcast_1153_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:36 INFO BlockManagerInfo: Added broadcast_1153_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:36 INFO SparkContext: Created broadcast 1153 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:36 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 763 (MapPartitionsRDD[2292] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:36 INFO TaskSchedulerImpl: Adding task set 763.0 with 8 tasks
16/09/07 19:11:36 INFO TaskSetManager: Starting task 0.0 in stage 763.0 (TID 6104, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 1.0 in stage 763.0 (TID 6105, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 2.0 in stage 763.0 (TID 6106, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 3.0 in stage 763.0 (TID 6107, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 4.0 in stage 763.0 (TID 6108, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 5.0 in stage 763.0 (TID 6109, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 6.0 in stage 763.0 (TID 6110, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:36 INFO TaskSetManager: Starting task 7.0 in stage 763.0 (TID 6111, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:36 INFO Executor: Running task 0.0 in stage 763.0 (TID 6104)
16/09/07 19:11:36 INFO Executor: Running task 4.0 in stage 763.0 (TID 6108)
16/09/07 19:11:36 INFO Executor: Running task 5.0 in stage 763.0 (TID 6109)
16/09/07 19:11:36 INFO Executor: Running task 1.0 in stage 763.0 (TID 6105)
16/09/07 19:11:36 INFO Executor: Running task 2.0 in stage 763.0 (TID 6106)
16/09/07 19:11:36 INFO Executor: Running task 3.0 in stage 763.0 (TID 6107)
16/09/07 19:11:36 INFO Executor: Running task 6.0 in stage 763.0 (TID 6110)
16/09/07 19:11:36 INFO Executor: Running task 7.0 in stage 763.0 (TID 6111)
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:36 INFO Executor: Finished task 2.0 in stage 763.0 (TID 6106). 2160 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 2.0 in stage 763.0 (TID 6106) in 1 ms on localhost (1/8)
16/09/07 19:11:36 INFO Executor: Finished task 1.0 in stage 763.0 (TID 6105). 2160 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 1.0 in stage 763.0 (TID 6105) in 2 ms on localhost (2/8)
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:36 INFO Executor: Finished task 4.0 in stage 763.0 (TID 6108). 2160 bytes result sent to driver
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:36 INFO TaskSetManager: Finished task 4.0 in stage 763.0 (TID 6108) in 2 ms on localhost (3/8)
16/09/07 19:11:36 INFO Executor: Finished task 3.0 in stage 763.0 (TID 6107). 2160 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 3.0 in stage 763.0 (TID 6107) in 2 ms on localhost (4/8)
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:36 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:36 INFO Executor: Finished task 5.0 in stage 763.0 (TID 6109). 2160 bytes result sent to driver
16/09/07 19:11:36 INFO TaskSetManager: Finished task 5.0 in stage 763.0 (TID 6109) in 3 ms on localhost (5/8)
16/09/07 19:11:36 INFO BlockManagerInfo: Removed broadcast_1152_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:36 INFO BlockManagerInfo: Removed broadcast_1150_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:36 INFO ContextCleaner: Cleaned accumulator 762
16/09/07 19:11:36 INFO ContextCleaner: Cleaned accumulator 761
16/09/07 19:11:36 INFO BlockManagerInfo: Removed broadcast_1148_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:36 INFO ContextCleaner: Cleaned shuffle 375
16/09/07 19:11:37 INFO Executor: Finished task 6.0 in stage 763.0 (TID 6110). 3485274 bytes result sent to driver
16/09/07 19:11:37 INFO Executor: Finished task 7.0 in stage 763.0 (TID 6111). 3485274 bytes result sent to driver
16/09/07 19:11:37 INFO Executor: Finished task 0.0 in stage 763.0 (TID 6104). 3485274 bytes result sent to driver
16/09/07 19:11:37 INFO TaskSetManager: Finished task 6.0 in stage 763.0 (TID 6110) in 848 ms on localhost (6/8)
16/09/07 19:11:37 INFO TaskSetManager: Finished task 7.0 in stage 763.0 (TID 6111) in 864 ms on localhost (7/8)
16/09/07 19:11:37 INFO TaskSetManager: Finished task 0.0 in stage 763.0 (TID 6104) in 865 ms on localhost (8/8)
16/09/07 19:11:37 INFO TaskSchedulerImpl: Removed TaskSet 763.0, whose tasks have all completed, from pool 
16/09/07 19:11:37 INFO DAGScheduler: ResultStage 763 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.865 s
16/09/07 19:11:37 INFO DAGScheduler: Job 386 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.872225 s
16/09/07 19:11:37 INFO ParameterAveragingTrainingMaster: Completed training of split 65 of 78
16/09/07 19:11:37 INFO ParameterAveragingTrainingMaster: Starting training of split 66 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:37 INFO BlockManagerInfo: Removed broadcast_1153_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:37 INFO ContextCleaner: Cleaned accumulator 764
16/09/07 19:11:37 INFO ContextCleaner: Cleaned accumulator 763
16/09/07 19:11:37 INFO BlockManagerInfo: Removed broadcast_1151_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:37 INFO ContextCleaner: Cleaned shuffle 376
16/09/07 19:11:37 INFO MemoryStore: ensureFreeSpace(19279640) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:37 INFO MemoryStore: Block broadcast_1154 stored as values in memory (estimated size 18.4 MB, free 3.7 GB)
16/09/07 19:11:37 INFO MemoryStore: ensureFreeSpace(167968) called with curMem=19823464, maxMem=4042889625
16/09/07 19:11:37 INFO MemoryStore: Block broadcast_1154_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:37 INFO BlockManagerInfo: Added broadcast_1154_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:37 INFO SparkContext: Created broadcast 1154 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:37 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:37 INFO DAGScheduler: Registering RDD 2293 (repartition at SparkUtils.java:190)
16/09/07 19:11:37 INFO DAGScheduler: Got job 387 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:37 INFO DAGScheduler: Final stage: ResultStage 765(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:37 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 764)
16/09/07 19:11:37 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 764)
16/09/07 19:11:37 INFO DAGScheduler: Submitting ShuffleMapStage 764 (MapPartitionsRDD[2293] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:37 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19991432, maxMem=4042889625
16/09/07 19:11:37 INFO MemoryStore: Block broadcast_1155 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:37 INFO MemoryStore: ensureFreeSpace(2563) called with curMem=19995896, maxMem=4042889625
16/09/07 19:11:37 INFO MemoryStore: Block broadcast_1155_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:37 INFO BlockManagerInfo: Added broadcast_1155_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:37 INFO SparkContext: Created broadcast 1155 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:37 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 764 (MapPartitionsRDD[2293] at repartition at SparkUtils.java:190)
16/09/07 19:11:37 INFO TaskSchedulerImpl: Adding task set 764.0 with 8 tasks
16/09/07 19:11:37 INFO TaskSetManager: Starting task 0.0 in stage 764.0 (TID 6112, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 1.0 in stage 764.0 (TID 6113, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 2.0 in stage 764.0 (TID 6114, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 3.0 in stage 764.0 (TID 6115, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 4.0 in stage 764.0 (TID 6116, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 5.0 in stage 764.0 (TID 6117, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 6.0 in stage 764.0 (TID 6118, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 7.0 in stage 764.0 (TID 6119, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:37 INFO Executor: Running task 0.0 in stage 764.0 (TID 6112)
16/09/07 19:11:37 INFO Executor: Running task 2.0 in stage 764.0 (TID 6114)
16/09/07 19:11:37 INFO Executor: Running task 6.0 in stage 764.0 (TID 6118)
16/09/07 19:11:37 INFO Executor: Running task 7.0 in stage 764.0 (TID 6119)
16/09/07 19:11:37 INFO Executor: Running task 5.0 in stage 764.0 (TID 6117)
16/09/07 19:11:37 INFO Executor: Running task 1.0 in stage 764.0 (TID 6113)
16/09/07 19:11:37 INFO Executor: Running task 4.0 in stage 764.0 (TID 6116)
16/09/07 19:11:37 INFO Executor: Running task 3.0 in stage 764.0 (TID 6115)
16/09/07 19:11:37 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:37 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:37 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:37 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:37 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:37 INFO Executor: Finished task 7.0 in stage 764.0 (TID 6119). 2260 bytes result sent to driver
16/09/07 19:11:37 INFO Executor: Finished task 6.0 in stage 764.0 (TID 6118). 2260 bytes result sent to driver
16/09/07 19:11:37 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:37 INFO TaskSetManager: Finished task 7.0 in stage 764.0 (TID 6119) in 2 ms on localhost (1/8)
16/09/07 19:11:37 INFO TaskSetManager: Finished task 6.0 in stage 764.0 (TID 6118) in 2 ms on localhost (2/8)
16/09/07 19:11:37 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:37 INFO Executor: Finished task 4.0 in stage 764.0 (TID 6116). 2260 bytes result sent to driver
16/09/07 19:11:37 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:37 INFO TaskSetManager: Finished task 4.0 in stage 764.0 (TID 6116) in 2 ms on localhost (3/8)
16/09/07 19:11:37 INFO Executor: Finished task 0.0 in stage 764.0 (TID 6112). 2260 bytes result sent to driver
16/09/07 19:11:37 INFO Executor: Finished task 2.0 in stage 764.0 (TID 6114). 2260 bytes result sent to driver
16/09/07 19:11:37 INFO Executor: Finished task 1.0 in stage 764.0 (TID 6113). 2260 bytes result sent to driver
16/09/07 19:11:37 INFO TaskSetManager: Finished task 0.0 in stage 764.0 (TID 6112) in 3 ms on localhost (4/8)
16/09/07 19:11:37 INFO TaskSetManager: Finished task 2.0 in stage 764.0 (TID 6114) in 3 ms on localhost (5/8)
16/09/07 19:11:37 INFO TaskSetManager: Finished task 1.0 in stage 764.0 (TID 6113) in 3 ms on localhost (6/8)
16/09/07 19:11:37 INFO Executor: Finished task 5.0 in stage 764.0 (TID 6117). 2260 bytes result sent to driver
16/09/07 19:11:37 INFO Executor: Finished task 3.0 in stage 764.0 (TID 6115). 2260 bytes result sent to driver
16/09/07 19:11:37 INFO TaskSetManager: Finished task 5.0 in stage 764.0 (TID 6117) in 3 ms on localhost (7/8)
16/09/07 19:11:37 INFO TaskSetManager: Finished task 3.0 in stage 764.0 (TID 6115) in 3 ms on localhost (8/8)
16/09/07 19:11:37 INFO TaskSchedulerImpl: Removed TaskSet 764.0, whose tasks have all completed, from pool 
16/09/07 19:11:37 INFO DAGScheduler: ShuffleMapStage 764 (repartition at SparkUtils.java:190) finished in 0.003 s
16/09/07 19:11:37 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:37 INFO DAGScheduler: running: Set()
16/09/07 19:11:37 INFO DAGScheduler: waiting: Set(ResultStage 765)
16/09/07 19:11:37 INFO DAGScheduler: failed: Set()
16/09/07 19:11:37 INFO DAGScheduler: Missing parents for ResultStage 765: List()
16/09/07 19:11:37 INFO DAGScheduler: Submitting ResultStage 765 (MapPartitionsRDD[2297] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:37 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19998459, maxMem=4042889625
16/09/07 19:11:37 INFO MemoryStore: Block broadcast_1156 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:37 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=20003555, maxMem=4042889625
16/09/07 19:11:37 INFO MemoryStore: Block broadcast_1156_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:37 INFO BlockManagerInfo: Added broadcast_1156_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:37 INFO SparkContext: Created broadcast 1156 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:37 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 765 (MapPartitionsRDD[2297] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:37 INFO TaskSchedulerImpl: Adding task set 765.0 with 8 tasks
16/09/07 19:11:37 INFO TaskSetManager: Starting task 0.0 in stage 765.0 (TID 6120, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 1.0 in stage 765.0 (TID 6121, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 2.0 in stage 765.0 (TID 6122, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 3.0 in stage 765.0 (TID 6123, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 4.0 in stage 765.0 (TID 6124, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 5.0 in stage 765.0 (TID 6125, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 6.0 in stage 765.0 (TID 6126, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:37 INFO TaskSetManager: Starting task 7.0 in stage 765.0 (TID 6127, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:37 INFO Executor: Running task 2.0 in stage 765.0 (TID 6122)
16/09/07 19:11:37 INFO Executor: Running task 5.0 in stage 765.0 (TID 6125)
16/09/07 19:11:37 INFO Executor: Running task 0.0 in stage 765.0 (TID 6120)
16/09/07 19:11:37 INFO Executor: Running task 7.0 in stage 765.0 (TID 6127)
16/09/07 19:11:37 INFO Executor: Running task 3.0 in stage 765.0 (TID 6123)
16/09/07 19:11:37 INFO Executor: Running task 4.0 in stage 765.0 (TID 6124)
16/09/07 19:11:37 INFO Executor: Running task 1.0 in stage 765.0 (TID 6121)
16/09/07 19:11:37 INFO Executor: Running task 6.0 in stage 765.0 (TID 6126)
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:37 INFO Executor: Finished task 2.0 in stage 765.0 (TID 6122). 2160 bytes result sent to driver
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:37 INFO TaskSetManager: Finished task 2.0 in stage 765.0 (TID 6122) in 1 ms on localhost (1/8)
16/09/07 19:11:37 INFO Executor: Finished task 1.0 in stage 765.0 (TID 6121). 2160 bytes result sent to driver
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:37 INFO TaskSetManager: Finished task 1.0 in stage 765.0 (TID 6121) in 2 ms on localhost (2/8)
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:37 INFO Executor: Finished task 4.0 in stage 765.0 (TID 6124). 2160 bytes result sent to driver
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:37 INFO TaskSetManager: Finished task 4.0 in stage 765.0 (TID 6124) in 2 ms on localhost (3/8)
16/09/07 19:11:37 INFO Executor: Finished task 3.0 in stage 765.0 (TID 6123). 2160 bytes result sent to driver
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:37 INFO TaskSetManager: Finished task 3.0 in stage 765.0 (TID 6123) in 2 ms on localhost (4/8)
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:37 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:37 INFO Executor: Finished task 5.0 in stage 765.0 (TID 6125). 2160 bytes result sent to driver
16/09/07 19:11:37 INFO TaskSetManager: Finished task 5.0 in stage 765.0 (TID 6125) in 3 ms on localhost (5/8)
16/09/07 19:11:38 INFO BlockManagerInfo: Removed broadcast_1155_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:38 INFO Executor: Finished task 0.0 in stage 765.0 (TID 6120). 3485274 bytes result sent to driver
16/09/07 19:11:38 INFO Executor: Finished task 7.0 in stage 765.0 (TID 6127). 3485274 bytes result sent to driver
16/09/07 19:11:38 INFO Executor: Finished task 6.0 in stage 765.0 (TID 6126). 3485274 bytes result sent to driver
16/09/07 19:11:38 INFO TaskSetManager: Finished task 0.0 in stage 765.0 (TID 6120) in 868 ms on localhost (6/8)
16/09/07 19:11:38 INFO TaskSetManager: Finished task 7.0 in stage 765.0 (TID 6127) in 871 ms on localhost (7/8)
16/09/07 19:11:38 INFO TaskSetManager: Finished task 6.0 in stage 765.0 (TID 6126) in 872 ms on localhost (8/8)
16/09/07 19:11:38 INFO DAGScheduler: ResultStage 765 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.872 s
16/09/07 19:11:38 INFO TaskSchedulerImpl: Removed TaskSet 765.0, whose tasks have all completed, from pool 
16/09/07 19:11:38 INFO DAGScheduler: Job 387 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.879285 s
16/09/07 19:11:38 INFO ParameterAveragingTrainingMaster: Completed training of split 66 of 78
16/09/07 19:11:38 INFO ParameterAveragingTrainingMaster: Starting training of split 67 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:38 INFO MemoryStore: ensureFreeSpace(19293008) called with curMem=19999272, maxMem=4042889625
16/09/07 19:11:38 INFO MemoryStore: Block broadcast_1157 stored as values in memory (estimated size 18.4 MB, free 3.7 GB)
16/09/07 19:11:39 INFO BlockManagerInfo: Removed broadcast_1156_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:39 INFO ContextCleaner: Cleaned accumulator 766
16/09/07 19:11:39 INFO ContextCleaner: Cleaned accumulator 765
16/09/07 19:11:39 INFO BlockManagerInfo: Removed broadcast_1154_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:39 INFO ContextCleaner: Cleaned shuffle 377
16/09/07 19:11:39 INFO MemoryStore: ensureFreeSpace(167953) called with curMem=19836832, maxMem=4042889625
16/09/07 19:11:39 INFO MemoryStore: Block broadcast_1157_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:39 INFO BlockManagerInfo: Added broadcast_1157_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:39 INFO SparkContext: Created broadcast 1157 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:39 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:39 INFO DAGScheduler: Registering RDD 2298 (repartition at SparkUtils.java:190)
16/09/07 19:11:39 INFO DAGScheduler: Got job 388 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:39 INFO DAGScheduler: Final stage: ResultStage 767(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:39 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 766)
16/09/07 19:11:39 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 766)
16/09/07 19:11:39 INFO DAGScheduler: Submitting ShuffleMapStage 766 (MapPartitionsRDD[2298] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:39 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=20004785, maxMem=4042889625
16/09/07 19:11:39 INFO MemoryStore: Block broadcast_1158 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:39 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=20009249, maxMem=4042889625
16/09/07 19:11:39 INFO MemoryStore: Block broadcast_1158_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:39 INFO BlockManagerInfo: Added broadcast_1158_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:39 INFO SparkContext: Created broadcast 1158 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:39 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 766 (MapPartitionsRDD[2298] at repartition at SparkUtils.java:190)
16/09/07 19:11:39 INFO TaskSchedulerImpl: Adding task set 766.0 with 8 tasks
16/09/07 19:11:39 INFO TaskSetManager: Starting task 0.0 in stage 766.0 (TID 6128, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 1.0 in stage 766.0 (TID 6129, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 2.0 in stage 766.0 (TID 6130, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 3.0 in stage 766.0 (TID 6131, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 4.0 in stage 766.0 (TID 6132, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 5.0 in stage 766.0 (TID 6133, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 6.0 in stage 766.0 (TID 6134, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 7.0 in stage 766.0 (TID 6135, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:39 INFO Executor: Running task 2.0 in stage 766.0 (TID 6130)
16/09/07 19:11:39 INFO Executor: Running task 0.0 in stage 766.0 (TID 6128)
16/09/07 19:11:39 INFO Executor: Running task 3.0 in stage 766.0 (TID 6131)
16/09/07 19:11:39 INFO Executor: Running task 1.0 in stage 766.0 (TID 6129)
16/09/07 19:11:39 INFO Executor: Running task 5.0 in stage 766.0 (TID 6133)
16/09/07 19:11:39 INFO Executor: Running task 4.0 in stage 766.0 (TID 6132)
16/09/07 19:11:39 INFO Executor: Running task 6.0 in stage 766.0 (TID 6134)
16/09/07 19:11:39 INFO Executor: Running task 7.0 in stage 766.0 (TID 6135)
16/09/07 19:11:39 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:39 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:39 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:39 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:39 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:39 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:39 INFO Executor: Finished task 0.0 in stage 766.0 (TID 6128). 2260 bytes result sent to driver
16/09/07 19:11:39 INFO Executor: Finished task 7.0 in stage 766.0 (TID 6135). 2260 bytes result sent to driver
16/09/07 19:11:39 INFO Executor: Finished task 2.0 in stage 766.0 (TID 6130). 2260 bytes result sent to driver
16/09/07 19:11:39 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:39 INFO Executor: Finished task 6.0 in stage 766.0 (TID 6134). 2260 bytes result sent to driver
16/09/07 19:11:39 INFO Executor: Finished task 3.0 in stage 766.0 (TID 6131). 2260 bytes result sent to driver
16/09/07 19:11:39 INFO TaskSetManager: Finished task 6.0 in stage 766.0 (TID 6134) in 3 ms on localhost (1/8)
16/09/07 19:11:39 INFO TaskSetManager: Finished task 2.0 in stage 766.0 (TID 6130) in 3 ms on localhost (2/8)
16/09/07 19:11:39 INFO TaskSetManager: Finished task 7.0 in stage 766.0 (TID 6135) in 3 ms on localhost (3/8)
16/09/07 19:11:39 INFO TaskSetManager: Finished task 0.0 in stage 766.0 (TID 6128) in 3 ms on localhost (4/8)
16/09/07 19:11:39 INFO TaskSetManager: Finished task 3.0 in stage 766.0 (TID 6131) in 3 ms on localhost (5/8)
16/09/07 19:11:39 INFO Executor: Finished task 4.0 in stage 766.0 (TID 6132). 2260 bytes result sent to driver
16/09/07 19:11:39 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:39 INFO TaskSetManager: Finished task 4.0 in stage 766.0 (TID 6132) in 3 ms on localhost (6/8)
16/09/07 19:11:39 INFO Executor: Finished task 5.0 in stage 766.0 (TID 6133). 2260 bytes result sent to driver
16/09/07 19:11:39 INFO Executor: Finished task 1.0 in stage 766.0 (TID 6129). 2260 bytes result sent to driver
16/09/07 19:11:39 INFO TaskSetManager: Finished task 5.0 in stage 766.0 (TID 6133) in 4 ms on localhost (7/8)
16/09/07 19:11:39 INFO TaskSetManager: Finished task 1.0 in stage 766.0 (TID 6129) in 4 ms on localhost (8/8)
16/09/07 19:11:39 INFO TaskSchedulerImpl: Removed TaskSet 766.0, whose tasks have all completed, from pool 
16/09/07 19:11:39 INFO DAGScheduler: ShuffleMapStage 766 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:39 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:39 INFO DAGScheduler: running: Set()
16/09/07 19:11:39 INFO DAGScheduler: waiting: Set(ResultStage 767)
16/09/07 19:11:39 INFO DAGScheduler: failed: Set()
16/09/07 19:11:39 INFO DAGScheduler: Missing parents for ResultStage 767: List()
16/09/07 19:11:39 INFO DAGScheduler: Submitting ResultStage 767 (MapPartitionsRDD[2302] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:39 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=20011813, maxMem=4042889625
16/09/07 19:11:39 INFO MemoryStore: Block broadcast_1159 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:39 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=20016909, maxMem=4042889625
16/09/07 19:11:39 INFO MemoryStore: Block broadcast_1159_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:39 INFO BlockManagerInfo: Added broadcast_1159_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:39 INFO SparkContext: Created broadcast 1159 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:39 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 767 (MapPartitionsRDD[2302] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:39 INFO TaskSchedulerImpl: Adding task set 767.0 with 8 tasks
16/09/07 19:11:39 INFO TaskSetManager: Starting task 0.0 in stage 767.0 (TID 6136, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 1.0 in stage 767.0 (TID 6137, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 2.0 in stage 767.0 (TID 6138, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 3.0 in stage 767.0 (TID 6139, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 4.0 in stage 767.0 (TID 6140, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 5.0 in stage 767.0 (TID 6141, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 6.0 in stage 767.0 (TID 6142, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:39 INFO TaskSetManager: Starting task 7.0 in stage 767.0 (TID 6143, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:39 INFO Executor: Running task 1.0 in stage 767.0 (TID 6137)
16/09/07 19:11:39 INFO Executor: Running task 3.0 in stage 767.0 (TID 6139)
16/09/07 19:11:39 INFO Executor: Running task 2.0 in stage 767.0 (TID 6138)
16/09/07 19:11:39 INFO Executor: Running task 7.0 in stage 767.0 (TID 6143)
16/09/07 19:11:39 INFO Executor: Running task 4.0 in stage 767.0 (TID 6140)
16/09/07 19:11:39 INFO Executor: Running task 0.0 in stage 767.0 (TID 6136)
16/09/07 19:11:39 INFO Executor: Running task 6.0 in stage 767.0 (TID 6142)
16/09/07 19:11:39 INFO Executor: Running task 5.0 in stage 767.0 (TID 6141)
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:39 INFO Executor: Finished task 5.0 in stage 767.0 (TID 6141). 2160 bytes result sent to driver
16/09/07 19:11:39 INFO TaskSetManager: Finished task 5.0 in stage 767.0 (TID 6141) in 1 ms on localhost (1/8)
16/09/07 19:11:39 INFO Executor: Finished task 4.0 in stage 767.0 (TID 6140). 2160 bytes result sent to driver
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:39 INFO TaskSetManager: Finished task 4.0 in stage 767.0 (TID 6140) in 1 ms on localhost (2/8)
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:39 INFO Executor: Finished task 3.0 in stage 767.0 (TID 6139). 2160 bytes result sent to driver
16/09/07 19:11:39 INFO TaskSetManager: Finished task 3.0 in stage 767.0 (TID 6139) in 2 ms on localhost (3/8)
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:39 INFO Executor: Finished task 2.0 in stage 767.0 (TID 6138). 2160 bytes result sent to driver
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:39 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:39 INFO TaskSetManager: Finished task 2.0 in stage 767.0 (TID 6138) in 3 ms on localhost (4/8)
16/09/07 19:11:39 INFO Executor: Finished task 1.0 in stage 767.0 (TID 6137). 2160 bytes result sent to driver
16/09/07 19:11:39 INFO TaskSetManager: Finished task 1.0 in stage 767.0 (TID 6137) in 3 ms on localhost (5/8)
16/09/07 19:11:39 INFO BlockManagerInfo: Removed broadcast_1158_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:39 INFO Executor: Finished task 6.0 in stage 767.0 (TID 6142). 3485274 bytes result sent to driver
16/09/07 19:11:39 INFO Executor: Finished task 0.0 in stage 767.0 (TID 6136). 3485274 bytes result sent to driver
16/09/07 19:11:39 INFO Executor: Finished task 7.0 in stage 767.0 (TID 6143). 3485274 bytes result sent to driver
16/09/07 19:11:39 INFO TaskSetManager: Finished task 6.0 in stage 767.0 (TID 6142) in 894 ms on localhost (6/8)
16/09/07 19:11:39 INFO TaskSetManager: Finished task 0.0 in stage 767.0 (TID 6136) in 894 ms on localhost (7/8)
16/09/07 19:11:39 INFO TaskSetManager: Finished task 7.0 in stage 767.0 (TID 6143) in 894 ms on localhost (8/8)
16/09/07 19:11:39 INFO TaskSchedulerImpl: Removed TaskSet 767.0, whose tasks have all completed, from pool 
16/09/07 19:11:39 INFO DAGScheduler: ResultStage 767 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.894 s
16/09/07 19:11:39 INFO DAGScheduler: Job 388 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.903162 s
16/09/07 19:11:39 INFO ParameterAveragingTrainingMaster: Completed training of split 67 of 78
16/09/07 19:11:39 INFO ParameterAveragingTrainingMaster: Starting training of split 68 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:40 INFO MemoryStore: ensureFreeSpace(19504864) called with curMem=20012624, maxMem=4042889625
16/09/07 19:11:40 INFO MemoryStore: Block broadcast_1160 stored as values in memory (estimated size 18.6 MB, free 3.7 GB)
16/09/07 19:11:40 INFO MemoryStore: ensureFreeSpace(168008) called with curMem=39517488, maxMem=4042889625
16/09/07 19:11:40 INFO MemoryStore: Block broadcast_1160_piece0 stored as bytes in memory (estimated size 164.1 KB, free 3.7 GB)
16/09/07 19:11:40 INFO BlockManagerInfo: Added broadcast_1160_piece0 in memory on localhost:41251 (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:40 INFO SparkContext: Created broadcast 1160 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:40 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:40 INFO DAGScheduler: Registering RDD 2303 (repartition at SparkUtils.java:190)
16/09/07 19:11:40 INFO DAGScheduler: Got job 389 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:40 INFO DAGScheduler: Final stage: ResultStage 769(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:40 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 768)
16/09/07 19:11:40 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 768)
16/09/07 19:11:40 INFO DAGScheduler: Submitting ShuffleMapStage 768 (MapPartitionsRDD[2303] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:40 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=39685496, maxMem=4042889625
16/09/07 19:11:40 INFO MemoryStore: Block broadcast_1161 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:40 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=39689960, maxMem=4042889625
16/09/07 19:11:40 INFO MemoryStore: Block broadcast_1161_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:40 INFO BlockManagerInfo: Added broadcast_1161_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:40 INFO SparkContext: Created broadcast 1161 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:40 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 768 (MapPartitionsRDD[2303] at repartition at SparkUtils.java:190)
16/09/07 19:11:40 INFO TaskSchedulerImpl: Adding task set 768.0 with 8 tasks
16/09/07 19:11:40 INFO TaskSetManager: Starting task 0.0 in stage 768.0 (TID 6144, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 1.0 in stage 768.0 (TID 6145, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 2.0 in stage 768.0 (TID 6146, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 3.0 in stage 768.0 (TID 6147, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 4.0 in stage 768.0 (TID 6148, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 5.0 in stage 768.0 (TID 6149, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 6.0 in stage 768.0 (TID 6150, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 7.0 in stage 768.0 (TID 6151, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:40 INFO Executor: Running task 0.0 in stage 768.0 (TID 6144)
16/09/07 19:11:40 INFO Executor: Running task 1.0 in stage 768.0 (TID 6145)
16/09/07 19:11:40 INFO Executor: Running task 5.0 in stage 768.0 (TID 6149)
16/09/07 19:11:40 INFO Executor: Running task 4.0 in stage 768.0 (TID 6148)
16/09/07 19:11:40 INFO Executor: Running task 6.0 in stage 768.0 (TID 6150)
16/09/07 19:11:40 INFO Executor: Running task 7.0 in stage 768.0 (TID 6151)
16/09/07 19:11:40 INFO Executor: Running task 3.0 in stage 768.0 (TID 6147)
16/09/07 19:11:40 INFO Executor: Running task 2.0 in stage 768.0 (TID 6146)
16/09/07 19:11:40 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:40 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:40 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:40 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:40 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:40 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:40 INFO Executor: Finished task 2.0 in stage 768.0 (TID 6146). 2260 bytes result sent to driver
16/09/07 19:11:40 INFO Executor: Finished task 6.0 in stage 768.0 (TID 6150). 2260 bytes result sent to driver
16/09/07 19:11:40 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:40 INFO TaskSetManager: Finished task 2.0 in stage 768.0 (TID 6146) in 2 ms on localhost (1/8)
16/09/07 19:11:40 INFO TaskSetManager: Finished task 6.0 in stage 768.0 (TID 6150) in 2 ms on localhost (2/8)
16/09/07 19:11:40 INFO Executor: Finished task 5.0 in stage 768.0 (TID 6149). 2260 bytes result sent to driver
16/09/07 19:11:40 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:40 INFO TaskSetManager: Finished task 5.0 in stage 768.0 (TID 6149) in 2 ms on localhost (3/8)
16/09/07 19:11:40 INFO Executor: Finished task 7.0 in stage 768.0 (TID 6151). 2260 bytes result sent to driver
16/09/07 19:11:40 INFO Executor: Finished task 3.0 in stage 768.0 (TID 6147). 2260 bytes result sent to driver
16/09/07 19:11:40 INFO TaskSetManager: Finished task 7.0 in stage 768.0 (TID 6151) in 2 ms on localhost (4/8)
16/09/07 19:11:40 INFO TaskSetManager: Finished task 3.0 in stage 768.0 (TID 6147) in 2 ms on localhost (5/8)
16/09/07 19:11:40 INFO Executor: Finished task 0.0 in stage 768.0 (TID 6144). 2260 bytes result sent to driver
16/09/07 19:11:40 INFO Executor: Finished task 4.0 in stage 768.0 (TID 6148). 2260 bytes result sent to driver
16/09/07 19:11:40 INFO TaskSetManager: Finished task 0.0 in stage 768.0 (TID 6144) in 4 ms on localhost (6/8)
16/09/07 19:11:40 INFO Executor: Finished task 1.0 in stage 768.0 (TID 6145). 2260 bytes result sent to driver
16/09/07 19:11:40 INFO TaskSetManager: Finished task 4.0 in stage 768.0 (TID 6148) in 3 ms on localhost (7/8)
16/09/07 19:11:40 INFO TaskSetManager: Finished task 1.0 in stage 768.0 (TID 6145) in 3 ms on localhost (8/8)
16/09/07 19:11:40 INFO TaskSchedulerImpl: Removed TaskSet 768.0, whose tasks have all completed, from pool 
16/09/07 19:11:40 INFO DAGScheduler: ShuffleMapStage 768 (repartition at SparkUtils.java:190) finished in 0.004 s
16/09/07 19:11:40 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:40 INFO DAGScheduler: running: Set()
16/09/07 19:11:40 INFO DAGScheduler: waiting: Set(ResultStage 769)
16/09/07 19:11:40 INFO DAGScheduler: failed: Set()
16/09/07 19:11:40 INFO DAGScheduler: Missing parents for ResultStage 769: List()
16/09/07 19:11:40 INFO DAGScheduler: Submitting ResultStage 769 (MapPartitionsRDD[2307] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:40 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=39692524, maxMem=4042889625
16/09/07 19:11:40 INFO MemoryStore: Block broadcast_1162 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:40 INFO MemoryStore: ensureFreeSpace(2744) called with curMem=39697620, maxMem=4042889625
16/09/07 19:11:40 INFO MemoryStore: Block broadcast_1162_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:40 INFO BlockManagerInfo: Added broadcast_1162_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:40 INFO SparkContext: Created broadcast 1162 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:40 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 769 (MapPartitionsRDD[2307] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:40 INFO TaskSchedulerImpl: Adding task set 769.0 with 8 tasks
16/09/07 19:11:40 INFO TaskSetManager: Starting task 0.0 in stage 769.0 (TID 6152, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 1.0 in stage 769.0 (TID 6153, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 2.0 in stage 769.0 (TID 6154, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:40 INFO BlockManagerInfo: Removed broadcast_1159_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 3.0 in stage 769.0 (TID 6155, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 4.0 in stage 769.0 (TID 6156, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 5.0 in stage 769.0 (TID 6157, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:40 INFO ContextCleaner: Cleaned accumulator 768
16/09/07 19:11:40 INFO ContextCleaner: Cleaned accumulator 767
16/09/07 19:11:40 INFO TaskSetManager: Starting task 6.0 in stage 769.0 (TID 6158, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:40 INFO TaskSetManager: Starting task 7.0 in stage 769.0 (TID 6159, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:40 INFO Executor: Running task 1.0 in stage 769.0 (TID 6153)
16/09/07 19:11:40 INFO Executor: Running task 2.0 in stage 769.0 (TID 6154)
16/09/07 19:11:40 INFO Executor: Running task 3.0 in stage 769.0 (TID 6155)
16/09/07 19:11:40 INFO Executor: Running task 0.0 in stage 769.0 (TID 6152)
16/09/07 19:11:40 INFO Executor: Running task 5.0 in stage 769.0 (TID 6157)
16/09/07 19:11:40 INFO BlockManagerInfo: Removed broadcast_1157_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:40 INFO ContextCleaner: Cleaned shuffle 378
16/09/07 19:11:40 INFO Executor: Running task 4.0 in stage 769.0 (TID 6156)
16/09/07 19:11:40 INFO Executor: Running task 6.0 in stage 769.0 (TID 6158)
16/09/07 19:11:40 INFO Executor: Running task 7.0 in stage 769.0 (TID 6159)
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:40 INFO Executor: Finished task 3.0 in stage 769.0 (TID 6155). 2160 bytes result sent to driver
16/09/07 19:11:40 INFO Executor: Finished task 4.0 in stage 769.0 (TID 6156). 2160 bytes result sent to driver
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:40 INFO TaskSetManager: Finished task 3.0 in stage 769.0 (TID 6155) in 1 ms on localhost (1/8)
16/09/07 19:11:40 INFO TaskSetManager: Finished task 4.0 in stage 769.0 (TID 6156) in 1 ms on localhost (2/8)
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:40 INFO Executor: Finished task 5.0 in stage 769.0 (TID 6157). 2160 bytes result sent to driver
16/09/07 19:11:40 INFO Executor: Finished task 1.0 in stage 769.0 (TID 6153). 2160 bytes result sent to driver
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:40 INFO TaskSetManager: Finished task 5.0 in stage 769.0 (TID 6157) in 2 ms on localhost (3/8)
16/09/07 19:11:40 INFO TaskSetManager: Finished task 1.0 in stage 769.0 (TID 6153) in 2 ms on localhost (4/8)
16/09/07 19:11:40 INFO Executor: Finished task 2.0 in stage 769.0 (TID 6154). 2160 bytes result sent to driver
16/09/07 19:11:40 INFO TaskSetManager: Finished task 2.0 in stage 769.0 (TID 6154) in 2 ms on localhost (5/8)
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:40 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:40 INFO BlockManagerInfo: Removed broadcast_1161_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:40 INFO Executor: Finished task 6.0 in stage 769.0 (TID 6158). 3485274 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 6.0 in stage 769.0 (TID 6158) in 786 ms on localhost (6/8)
16/09/07 19:11:41 INFO Executor: Finished task 7.0 in stage 769.0 (TID 6159). 3485274 bytes result sent to driver
16/09/07 19:11:41 INFO Executor: Finished task 0.0 in stage 769.0 (TID 6152). 3485274 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 7.0 in stage 769.0 (TID 6159) in 856 ms on localhost (7/8)
16/09/07 19:11:41 INFO TaskSetManager: Finished task 0.0 in stage 769.0 (TID 6152) in 857 ms on localhost (8/8)
16/09/07 19:11:41 INFO TaskSchedulerImpl: Removed TaskSet 769.0, whose tasks have all completed, from pool 
16/09/07 19:11:41 INFO DAGScheduler: ResultStage 769 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.858 s
16/09/07 19:11:41 INFO DAGScheduler: Job 389 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.937887 s
16/09/07 19:11:41 INFO ParameterAveragingTrainingMaster: Completed training of split 68 of 78
16/09/07 19:11:41 INFO ParameterAveragingTrainingMaster: Starting training of split 69 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:41 INFO BlockManagerInfo: Removed broadcast_1162_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:41 INFO ContextCleaner: Cleaned accumulator 770
16/09/07 19:11:41 INFO ContextCleaner: Cleaned accumulator 769
16/09/07 19:11:41 INFO BlockManagerInfo: Removed broadcast_1160_piece0 on localhost:41251 in memory (size: 164.1 KB, free: 3.8 GB)
16/09/07 19:11:41 INFO ContextCleaner: Cleaned shuffle 379
16/09/07 19:11:41 INFO MemoryStore: ensureFreeSpace(19150808) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:41 INFO MemoryStore: Block broadcast_1163 stored as values in memory (estimated size 18.3 MB, free 3.7 GB)
16/09/07 19:11:41 INFO MemoryStore: ensureFreeSpace(167967) called with curMem=19694632, maxMem=4042889625
16/09/07 19:11:41 INFO MemoryStore: Block broadcast_1163_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:41 INFO BlockManagerInfo: Added broadcast_1163_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:41 INFO SparkContext: Created broadcast 1163 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:41 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:41 INFO DAGScheduler: Registering RDD 2308 (repartition at SparkUtils.java:190)
16/09/07 19:11:41 INFO DAGScheduler: Got job 390 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:41 INFO DAGScheduler: Final stage: ResultStage 771(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:41 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 770)
16/09/07 19:11:41 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 770)
16/09/07 19:11:41 INFO DAGScheduler: Submitting ShuffleMapStage 770 (MapPartitionsRDD[2308] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:41 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19862599, maxMem=4042889625
16/09/07 19:11:41 INFO MemoryStore: Block broadcast_1164 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:41 INFO MemoryStore: ensureFreeSpace(2565) called with curMem=19867063, maxMem=4042889625
16/09/07 19:11:41 INFO MemoryStore: Block broadcast_1164_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:41 INFO BlockManagerInfo: Added broadcast_1164_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:41 INFO SparkContext: Created broadcast 1164 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:41 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 770 (MapPartitionsRDD[2308] at repartition at SparkUtils.java:190)
16/09/07 19:11:41 INFO TaskSchedulerImpl: Adding task set 770.0 with 8 tasks
16/09/07 19:11:41 INFO TaskSetManager: Starting task 0.0 in stage 770.0 (TID 6160, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 1.0 in stage 770.0 (TID 6161, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 2.0 in stage 770.0 (TID 6162, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 3.0 in stage 770.0 (TID 6163, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 4.0 in stage 770.0 (TID 6164, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 5.0 in stage 770.0 (TID 6165, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 6.0 in stage 770.0 (TID 6166, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 7.0 in stage 770.0 (TID 6167, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:41 INFO Executor: Running task 1.0 in stage 770.0 (TID 6161)
16/09/07 19:11:41 INFO Executor: Running task 0.0 in stage 770.0 (TID 6160)
16/09/07 19:11:41 INFO Executor: Running task 7.0 in stage 770.0 (TID 6167)
16/09/07 19:11:41 INFO Executor: Running task 4.0 in stage 770.0 (TID 6164)
16/09/07 19:11:41 INFO Executor: Running task 5.0 in stage 770.0 (TID 6165)
16/09/07 19:11:41 INFO Executor: Running task 2.0 in stage 770.0 (TID 6162)
16/09/07 19:11:41 INFO Executor: Running task 3.0 in stage 770.0 (TID 6163)
16/09/07 19:11:41 INFO Executor: Running task 6.0 in stage 770.0 (TID 6166)
16/09/07 19:11:41 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:41 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:41 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:41 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:41 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:41 INFO Executor: Finished task 0.0 in stage 770.0 (TID 6160). 2260 bytes result sent to driver
16/09/07 19:11:41 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:41 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:41 INFO TaskSetManager: Finished task 0.0 in stage 770.0 (TID 6160) in 2 ms on localhost (1/8)
16/09/07 19:11:41 INFO Executor: Finished task 3.0 in stage 770.0 (TID 6163). 2260 bytes result sent to driver
16/09/07 19:11:41 INFO Executor: Finished task 6.0 in stage 770.0 (TID 6166). 2260 bytes result sent to driver
16/09/07 19:11:41 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:41 INFO TaskSetManager: Finished task 6.0 in stage 770.0 (TID 6166) in 2 ms on localhost (2/8)
16/09/07 19:11:41 INFO TaskSetManager: Finished task 3.0 in stage 770.0 (TID 6163) in 2 ms on localhost (3/8)
16/09/07 19:11:41 INFO Executor: Finished task 4.0 in stage 770.0 (TID 6164). 2260 bytes result sent to driver
16/09/07 19:11:41 INFO Executor: Finished task 5.0 in stage 770.0 (TID 6165). 2260 bytes result sent to driver
16/09/07 19:11:41 INFO Executor: Finished task 2.0 in stage 770.0 (TID 6162). 2260 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 4.0 in stage 770.0 (TID 6164) in 3 ms on localhost (4/8)
16/09/07 19:11:41 INFO Executor: Finished task 7.0 in stage 770.0 (TID 6167). 2260 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 5.0 in stage 770.0 (TID 6165) in 3 ms on localhost (5/8)
16/09/07 19:11:41 INFO TaskSetManager: Finished task 2.0 in stage 770.0 (TID 6162) in 3 ms on localhost (6/8)
16/09/07 19:11:41 INFO Executor: Finished task 1.0 in stage 770.0 (TID 6161). 2260 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 7.0 in stage 770.0 (TID 6167) in 3 ms on localhost (7/8)
16/09/07 19:11:41 INFO TaskSetManager: Finished task 1.0 in stage 770.0 (TID 6161) in 3 ms on localhost (8/8)
16/09/07 19:11:41 INFO TaskSchedulerImpl: Removed TaskSet 770.0, whose tasks have all completed, from pool 
16/09/07 19:11:41 INFO DAGScheduler: ShuffleMapStage 770 (repartition at SparkUtils.java:190) finished in 0.003 s
16/09/07 19:11:41 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:41 INFO DAGScheduler: running: Set()
16/09/07 19:11:41 INFO DAGScheduler: waiting: Set(ResultStage 771)
16/09/07 19:11:41 INFO DAGScheduler: failed: Set()
16/09/07 19:11:41 INFO DAGScheduler: Missing parents for ResultStage 771: List()
16/09/07 19:11:41 INFO DAGScheduler: Submitting ResultStage 771 (MapPartitionsRDD[2312] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:41 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19869628, maxMem=4042889625
16/09/07 19:11:41 INFO MemoryStore: Block broadcast_1165 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:41 INFO MemoryStore: ensureFreeSpace(2739) called with curMem=19874724, maxMem=4042889625
16/09/07 19:11:41 INFO MemoryStore: Block broadcast_1165_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:41 INFO BlockManagerInfo: Added broadcast_1165_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:41 INFO SparkContext: Created broadcast 1165 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:41 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 771 (MapPartitionsRDD[2312] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:41 INFO TaskSchedulerImpl: Adding task set 771.0 with 8 tasks
16/09/07 19:11:41 INFO TaskSetManager: Starting task 0.0 in stage 771.0 (TID 6168, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 1.0 in stage 771.0 (TID 6169, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 2.0 in stage 771.0 (TID 6170, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 3.0 in stage 771.0 (TID 6171, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 4.0 in stage 771.0 (TID 6172, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 5.0 in stage 771.0 (TID 6173, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 6.0 in stage 771.0 (TID 6174, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:41 INFO TaskSetManager: Starting task 7.0 in stage 771.0 (TID 6175, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:41 INFO Executor: Running task 0.0 in stage 771.0 (TID 6168)
16/09/07 19:11:41 INFO Executor: Running task 1.0 in stage 771.0 (TID 6169)
16/09/07 19:11:41 INFO Executor: Running task 7.0 in stage 771.0 (TID 6175)
16/09/07 19:11:41 INFO Executor: Running task 2.0 in stage 771.0 (TID 6170)
16/09/07 19:11:41 INFO Executor: Running task 5.0 in stage 771.0 (TID 6173)
16/09/07 19:11:41 INFO Executor: Running task 4.0 in stage 771.0 (TID 6172)
16/09/07 19:11:41 INFO Executor: Running task 6.0 in stage 771.0 (TID 6174)
16/09/07 19:11:41 INFO Executor: Running task 3.0 in stage 771.0 (TID 6171)
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:41 INFO Executor: Finished task 2.0 in stage 771.0 (TID 6170). 2160 bytes result sent to driver
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:41 INFO Executor: Finished task 4.0 in stage 771.0 (TID 6172). 2160 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 2.0 in stage 771.0 (TID 6170) in 2 ms on localhost (1/8)
16/09/07 19:11:41 INFO Executor: Finished task 5.0 in stage 771.0 (TID 6173). 2160 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 4.0 in stage 771.0 (TID 6172) in 2 ms on localhost (2/8)
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:41 INFO TaskSetManager: Finished task 5.0 in stage 771.0 (TID 6173) in 2 ms on localhost (3/8)
16/09/07 19:11:41 INFO Executor: Finished task 1.0 in stage 771.0 (TID 6169). 2160 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 1.0 in stage 771.0 (TID 6169) in 2 ms on localhost (4/8)
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Getting 8 non-empty blocks out of 8 blocks
16/09/07 19:11:41 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
16/09/07 19:11:41 INFO Executor: Finished task 3.0 in stage 771.0 (TID 6171). 2160 bytes result sent to driver
16/09/07 19:11:41 INFO TaskSetManager: Finished task 3.0 in stage 771.0 (TID 6171) in 3 ms on localhost (5/8)
16/09/07 19:11:41 INFO BlockManagerInfo: Removed broadcast_1164_piece0 on localhost:41251 in memory (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:42 INFO Executor: Finished task 7.0 in stage 771.0 (TID 6175). 3485274 bytes result sent to driver
16/09/07 19:11:42 INFO TaskSetManager: Finished task 7.0 in stage 771.0 (TID 6175) in 884 ms on localhost (6/8)
16/09/07 19:11:42 INFO Executor: Finished task 0.0 in stage 771.0 (TID 6168). 3485524 bytes result sent to driver
16/09/07 19:11:42 INFO Executor: Finished task 6.0 in stage 771.0 (TID 6174). 3485524 bytes result sent to driver
16/09/07 19:11:42 INFO TaskSetManager: Finished task 0.0 in stage 771.0 (TID 6168) in 950 ms on localhost (7/8)
16/09/07 19:11:42 INFO TaskSetManager: Finished task 6.0 in stage 771.0 (TID 6174) in 950 ms on localhost (8/8)
16/09/07 19:11:42 INFO TaskSchedulerImpl: Removed TaskSet 771.0, whose tasks have all completed, from pool 
16/09/07 19:11:42 INFO DAGScheduler: ResultStage 771 (aggregate at ParameterAveragingTrainingMaster.java:612) finished in 0.950 s
16/09/07 19:11:42 INFO DAGScheduler: Job 390 finished: aggregate at ParameterAveragingTrainingMaster.java:612, took 0.958778 s
16/09/07 19:11:42 INFO ParameterAveragingTrainingMaster: Completed training of split 69 of 78
16/09/07 19:11:42 INFO ParameterAveragingTrainingMaster: Starting training of split 70 of 78. workerMiniBatchSize=32, averagingFreq=3, Configured for 8 workers
16/09/07 19:11:42 INFO BlockManagerInfo: Removed broadcast_1165_piece0 on localhost:41251 in memory (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:42 INFO ContextCleaner: Cleaned accumulator 772
16/09/07 19:11:42 INFO ContextCleaner: Cleaned accumulator 771
16/09/07 19:11:42 INFO BlockManagerInfo: Removed broadcast_1163_piece0 on localhost:41251 in memory (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:42 INFO ContextCleaner: Cleaned shuffle 380
16/09/07 19:11:42 INFO MemoryStore: ensureFreeSpace(19166608) called with curMem=543824, maxMem=4042889625
16/09/07 19:11:42 INFO MemoryStore: Block broadcast_1166 stored as values in memory (estimated size 18.3 MB, free 3.7 GB)
16/09/07 19:11:42 INFO MemoryStore: ensureFreeSpace(167963) called with curMem=19710432, maxMem=4042889625
16/09/07 19:11:42 INFO MemoryStore: Block broadcast_1166_piece0 stored as bytes in memory (estimated size 164.0 KB, free 3.7 GB)
16/09/07 19:11:42 INFO BlockManagerInfo: Added broadcast_1166_piece0 in memory on localhost:41251 (size: 164.0 KB, free: 3.8 GB)
16/09/07 19:11:42 INFO SparkContext: Created broadcast 1166 from broadcast at ParameterAveragingTrainingMaster.java:151
16/09/07 19:11:42 INFO SparkContext: Starting job: aggregate at ParameterAveragingTrainingMaster.java:612
16/09/07 19:11:42 INFO DAGScheduler: Registering RDD 2313 (repartition at SparkUtils.java:190)
16/09/07 19:11:42 INFO DAGScheduler: Got job 391 (aggregate at ParameterAveragingTrainingMaster.java:612) with 8 output partitions
16/09/07 19:11:42 INFO DAGScheduler: Final stage: ResultStage 773(aggregate at ParameterAveragingTrainingMaster.java:612)
16/09/07 19:11:42 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 772)
16/09/07 19:11:42 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 772)
16/09/07 19:11:42 INFO DAGScheduler: Submitting ShuffleMapStage 772 (MapPartitionsRDD[2313] at repartition at SparkUtils.java:190), which has no missing parents
16/09/07 19:11:42 INFO MemoryStore: ensureFreeSpace(4464) called with curMem=19878395, maxMem=4042889625
16/09/07 19:11:42 INFO MemoryStore: Block broadcast_1167 stored as values in memory (estimated size 4.4 KB, free 3.7 GB)
16/09/07 19:11:42 INFO MemoryStore: ensureFreeSpace(2564) called with curMem=19882859, maxMem=4042889625
16/09/07 19:11:42 INFO MemoryStore: Block broadcast_1167_piece0 stored as bytes in memory (estimated size 2.5 KB, free 3.7 GB)
16/09/07 19:11:42 INFO BlockManagerInfo: Added broadcast_1167_piece0 in memory on localhost:41251 (size: 2.5 KB, free: 3.8 GB)
16/09/07 19:11:42 INFO SparkContext: Created broadcast 1167 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:42 INFO DAGScheduler: Submitting 8 missing tasks from ShuffleMapStage 772 (MapPartitionsRDD[2313] at repartition at SparkUtils.java:190)
16/09/07 19:11:42 INFO TaskSchedulerImpl: Adding task set 772.0 with 8 tasks
16/09/07 19:11:42 INFO TaskSetManager: Starting task 0.0 in stage 772.0 (TID 6176, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 1.0 in stage 772.0 (TID 6177, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 2.0 in stage 772.0 (TID 6178, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 3.0 in stage 772.0 (TID 6179, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 4.0 in stage 772.0 (TID 6180, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 5.0 in stage 772.0 (TID 6181, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 6.0 in stage 772.0 (TID 6182, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 7.0 in stage 772.0 (TID 6183, localhost, PROCESS_LOCAL, 2091 bytes)
16/09/07 19:11:42 INFO Executor: Running task 2.0 in stage 772.0 (TID 6178)
16/09/07 19:11:42 INFO Executor: Running task 6.0 in stage 772.0 (TID 6182)
16/09/07 19:11:42 INFO Executor: Running task 4.0 in stage 772.0 (TID 6180)
16/09/07 19:11:42 INFO Executor: Running task 5.0 in stage 772.0 (TID 6181)
16/09/07 19:11:42 INFO Executor: Running task 0.0 in stage 772.0 (TID 6176)
16/09/07 19:11:42 INFO Executor: Running task 7.0 in stage 772.0 (TID 6183)
16/09/07 19:11:42 INFO Executor: Running task 3.0 in stage 772.0 (TID 6179)
16/09/07 19:11:42 INFO Executor: Running task 1.0 in stage 772.0 (TID 6177)
16/09/07 19:11:42 INFO BlockManager: Found block rdd_1889_3 locally
16/09/07 19:11:42 INFO BlockManager: Found block rdd_1889_7 locally
16/09/07 19:11:42 INFO Executor: Finished task 3.0 in stage 772.0 (TID 6179). 2260 bytes result sent to driver
16/09/07 19:11:42 INFO BlockManager: Found block rdd_1889_2 locally
16/09/07 19:11:42 INFO TaskSetManager: Finished task 3.0 in stage 772.0 (TID 6179) in 2 ms on localhost (1/8)
16/09/07 19:11:42 INFO Executor: Finished task 7.0 in stage 772.0 (TID 6183). 2260 bytes result sent to driver
16/09/07 19:11:42 INFO TaskSetManager: Finished task 7.0 in stage 772.0 (TID 6183) in 2 ms on localhost (2/8)
16/09/07 19:11:42 INFO Executor: Finished task 2.0 in stage 772.0 (TID 6178). 2260 bytes result sent to driver
16/09/07 19:11:42 INFO BlockManager: Found block rdd_1889_0 locally
16/09/07 19:11:42 INFO TaskSetManager: Finished task 2.0 in stage 772.0 (TID 6178) in 3 ms on localhost (3/8)
16/09/07 19:11:42 INFO BlockManager: Found block rdd_1889_6 locally
16/09/07 19:11:42 INFO BlockManager: Found block rdd_1889_1 locally
16/09/07 19:11:42 INFO BlockManager: Found block rdd_1889_4 locally
16/09/07 19:11:42 INFO Executor: Finished task 6.0 in stage 772.0 (TID 6182). 2260 bytes result sent to driver
16/09/07 19:11:42 INFO Executor: Finished task 0.0 in stage 772.0 (TID 6176). 2260 bytes result sent to driver
16/09/07 19:11:42 INFO BlockManager: Found block rdd_1889_5 locally
16/09/07 19:11:42 INFO TaskSetManager: Finished task 6.0 in stage 772.0 (TID 6182) in 4 ms on localhost (4/8)
16/09/07 19:11:42 INFO TaskSetManager: Finished task 0.0 in stage 772.0 (TID 6176) in 5 ms on localhost (5/8)
16/09/07 19:11:42 INFO Executor: Finished task 4.0 in stage 772.0 (TID 6180). 2260 bytes result sent to driver
16/09/07 19:11:42 INFO Executor: Finished task 1.0 in stage 772.0 (TID 6177). 2260 bytes result sent to driver
16/09/07 19:11:42 INFO TaskSetManager: Finished task 4.0 in stage 772.0 (TID 6180) in 4 ms on localhost (6/8)
16/09/07 19:11:42 INFO TaskSetManager: Finished task 1.0 in stage 772.0 (TID 6177) in 4 ms on localhost (7/8)
16/09/07 19:11:42 INFO Executor: Finished task 5.0 in stage 772.0 (TID 6181). 2260 bytes result sent to driver
16/09/07 19:11:42 INFO TaskSetManager: Finished task 5.0 in stage 772.0 (TID 6181) in 4 ms on localhost (8/8)
16/09/07 19:11:42 INFO TaskSchedulerImpl: Removed TaskSet 772.0, whose tasks have all completed, from pool 
16/09/07 19:11:42 INFO DAGScheduler: ShuffleMapStage 772 (repartition at SparkUtils.java:190) finished in 0.005 s
16/09/07 19:11:42 INFO DAGScheduler: looking for newly runnable stages
16/09/07 19:11:42 INFO DAGScheduler: running: Set()
16/09/07 19:11:42 INFO DAGScheduler: waiting: Set(ResultStage 773)
16/09/07 19:11:42 INFO DAGScheduler: failed: Set()
16/09/07 19:11:42 INFO DAGScheduler: Missing parents for ResultStage 773: List()
16/09/07 19:11:42 INFO DAGScheduler: Submitting ResultStage 773 (MapPartitionsRDD[2317] at mapPartitions at ParameterAveragingTrainingMaster.java:543), which is now runnable
16/09/07 19:11:42 INFO MemoryStore: ensureFreeSpace(5096) called with curMem=19885423, maxMem=4042889625
16/09/07 19:11:42 INFO MemoryStore: Block broadcast_1168 stored as values in memory (estimated size 5.0 KB, free 3.7 GB)
16/09/07 19:11:42 INFO MemoryStore: ensureFreeSpace(2743) called with curMem=19890519, maxMem=4042889625
16/09/07 19:11:42 INFO MemoryStore: Block broadcast_1168_piece0 stored as bytes in memory (estimated size 2.7 KB, free 3.7 GB)
16/09/07 19:11:42 INFO BlockManagerInfo: Added broadcast_1168_piece0 in memory on localhost:41251 (size: 2.7 KB, free: 3.8 GB)
16/09/07 19:11:42 INFO SparkContext: Created broadcast 1168 from broadcast at DAGScheduler.scala:861
16/09/07 19:11:42 INFO DAGScheduler: Submitting 8 missing tasks from ResultStage 773 (MapPartitionsRDD[2317] at mapPartitions at ParameterAveragingTrainingMaster.java:543)
16/09/07 19:11:42 INFO TaskSchedulerImpl: Adding task set 773.0 with 8 tasks
16/09/07 19:11:42 INFO TaskSetManager: Starting task 0.0 in stage 773.0 (TID 6184, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 1.0 in stage 773.0 (TID 6185, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 2.0 in stage 773.0 (TID 6186, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 3.0 in stage 773.0 (TID 6187, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task 4.0 in stage 773.0 (TID 6188, localhost, PROCESS_LOCAL, 2089 bytes)
16/09/07 19:11:42 INFO TaskSetManager: Starting task