Last active
October 18, 2017 08:29
-
-
Save sebge2emasphere/01d2759365288ddbd0a085d23e09e6c9 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2017-10-18 07:30:47 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 20 | |
| 2017-10-18 07:30:47 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 21 | |
| 2017-10-18 07:30:47 INFO Executor:54 - Running task 0.0 in stage 18.0 (TID 20) | |
| 2017-10-18 07:30:47 DEBUG Executor:58 - Task 20's epoch is 10 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting local block broadcast_18 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Block broadcast_18 was not found | |
| 2017-10-18 07:30:47 INFO TorrentBroadcast:54 - Started reading broadcast variable 18 | |
| 2017-10-18 07:30:47 DEBUG TorrentBroadcast:58 - Reading piece broadcast_18_piece0 of broadcast_18 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting local block broadcast_18_piece0 as bytes | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting remote block broadcast_18_piece0 | |
| 2017-10-18 07:30:47 INFO Executor:54 - Running task 1.0 in stage 18.0 (TID 21) | |
| 2017-10-18 07:30:47 DEBUG Executor:58 - Task 21's epoch is 10 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting remote block broadcast_18_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:47 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:47 INFO MemoryStore:54 - Block broadcast_18_piece0 stored as bytes in memory (estimated size 4.8 KB, free 365.3 MB) | |
| 2017-10-18 07:30:47 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_18_piece0 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Told master about block broadcast_18_piece0 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Put block broadcast_18_piece0 locally took 3 ms | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Putting block broadcast_18_piece0 without replication took 3 ms | |
| 2017-10-18 07:30:47 INFO TorrentBroadcast:54 - Reading broadcast variable 18 took 16 ms | |
| 2017-10-18 07:30:47 INFO MemoryStore:54 - Block broadcast_18 stored as values in memory (estimated size 8.7 KB, free 365.3 MB) | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Put block broadcast_18 locally took 1 ms | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Putting block broadcast_18 without replication took 1 ms | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting local block broadcast_18 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Level for block broadcast_18 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
| 2017-10-18 07:30:47 INFO NewHadoopRDD:54 - Input split: HBase table split(table name: flow, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"EhxJA4eZW2uHon2wa4JHdw==","stopRow":"EhxJA4eZW2uHon2wa4JHdw=>","batch":-1,"cacheBlocks":true,"totalColumns":8,"maxResultSize":-1,"families":{"d":["ba","bb","c","h"]},"caching":20000,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: EhxJA4eZW2uHon2wa4JHdw==TrvEBoE/NHtt5hddvafuOg==t+Q75vuTHmtk6jdLMQXYJA==, end row: EhxJA4eZW2uHon2wa4JHdw=>, region location: ip-xx.ec2.internal, encoded region name: 19207f08fce41ba5f363079376ac2e29) | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting local block broadcast_16 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Block broadcast_16 was not found | |
| 2017-10-18 07:30:47 INFO TorrentBroadcast:54 - Started reading broadcast variable 16 | |
| 2017-10-18 07:30:47 DEBUG TorrentBroadcast:58 - Reading piece broadcast_16_piece0 of broadcast_16 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting local block broadcast_16_piece0 as bytes | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting remote block broadcast_16_piece0 | |
| 2017-10-18 07:30:47 INFO NewHadoopRDD:54 - Input split: HBase table split(table name: flow, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"EhxJA4eZW2uHon2wa4JHdw==","stopRow":"EhxJA4eZW2uHon2wa4JHdw=>","batch":-1,"cacheBlocks":true,"totalColumns":8,"maxResultSize":-1,"families":{"d":["ba","bb","c","h"]},"caching":20000,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: EhxJA4eZW2uHon2wa4JHdw==, end row: EhxJA4eZW2uHon2wa4JHdw==TrvEBoE/NHtt5hddvafuOg==t+Q75vuTHmtk6jdLMQXYJA==, region location: ip-xx.ec2.internal, encoded region name: 24c1c7b25e545cccc44c52cb0428718e) | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting remote block broadcast_16_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:47 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:47 INFO MemoryStore:54 - Block broadcast_16_piece0 stored as bytes in memory (estimated size 31.8 KB, free 365.3 MB) | |
| 2017-10-18 07:30:47 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_16_piece0 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Told master about block broadcast_16_piece0 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Put block broadcast_16_piece0 locally took 2 ms | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Putting block broadcast_16_piece0 without replication took 3 ms | |
| 2017-10-18 07:30:47 INFO TorrentBroadcast:54 - Reading broadcast variable 16 took 30 ms | |
| 2017-10-18 07:30:47 INFO MemoryStore:54 - Block broadcast_16 stored as values in memory (estimated size 471.3 KB, free 364.8 MB) | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Put block broadcast_16 locally took 8 ms | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Putting block broadcast_16 without replication took 8 ms | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Getting local block broadcast_16 | |
| 2017-10-18 07:30:47 DEBUG BlockManager:58 - Level for block broadcast_16 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
| 2017-10-18 07:30:47 INFO RecoverableZooKeeper:120 - Process identifier=hconnection-0x11de9185 connecting to ZooKeeper ensemble=localhost:2181 | |
| 2017-10-18 07:30:47 INFO ZooKeeper:438 - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@200209c6 | |
| 2017-10-18 07:30:47 INFO RecoverableZooKeeper:120 - Process identifier=hconnection-0x7272be70 connecting to ZooKeeper ensemble=localhost:2181 | |
| 2017-10-18 07:30:47 INFO ZooKeeper:438 - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@68df6ec3 | |
| 2017-10-18 07:30:47 INFO ClientCnxn:975 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) | |
| 2017-10-18 07:30:47 INFO ClientCnxn:975 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) | |
| 2017-10-18 07:30:47 INFO ClientCnxn:852 - Socket connection established to localhost/127.0.0.1:2181, initiating session | |
| 2017-10-18 07:30:47 INFO ClientCnxn:852 - Socket connection established to localhost/127.0.0.1:2181, initiating session | |
| 2017-10-18 07:30:47 INFO ClientCnxn:1235 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15f2bd51ca50012, negotiated timeout = 40000 | |
| 2017-10-18 07:30:47 INFO ClientCnxn:1235 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15f2bd51ca50013, negotiated timeout = 40000 | |
| 2017-10-18 07:30:47 INFO TableInputFormatBase:178 - Input split length: 232 M bytes. | |
| 2017-10-18 07:30:47 INFO TableInputFormatBase:178 - Input split length: 232 M bytes. | |
| 2017-10-18 07:30:48 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 22 | |
| 2017-10-18 07:30:48 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 23 | |
| 2017-10-18 07:30:48 INFO Executor:54 - Running task 0.0 in stage 23.0 (TID 22) | |
| 2017-10-18 07:30:48 INFO Executor:54 - Running task 1.0 in stage 23.0 (TID 23) | |
| 2017-10-18 07:30:48 DEBUG Executor:58 - Task 22's epoch is 10 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting local block broadcast_19 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Block broadcast_19 was not found | |
| 2017-10-18 07:30:48 INFO TorrentBroadcast:54 - Started reading broadcast variable 19 | |
| 2017-10-18 07:30:48 DEBUG TorrentBroadcast:58 - Reading piece broadcast_19_piece0 of broadcast_19 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting local block broadcast_19_piece0 as bytes | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting remote block broadcast_19_piece0 | |
| 2017-10-18 07:30:48 DEBUG Executor:58 - Task 23's epoch is 10 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting remote block broadcast_19_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:48 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:48 INFO MemoryStore:54 - Block broadcast_19_piece0 stored as bytes in memory (estimated size 4.8 KB, free 364.8 MB) | |
| 2017-10-18 07:30:48 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_19_piece0 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Told master about block broadcast_19_piece0 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Put block broadcast_19_piece0 locally took 3 ms | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Putting block broadcast_19_piece0 without replication took 3 ms | |
| 2017-10-18 07:30:48 INFO TorrentBroadcast:54 - Reading broadcast variable 19 took 26 ms | |
| 2017-10-18 07:30:48 INFO MemoryStore:54 - Block broadcast_19 stored as values in memory (estimated size 8.7 KB, free 364.8 MB) | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Put block broadcast_19 locally took 1 ms | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Putting block broadcast_19 without replication took 1 ms | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting local block broadcast_19 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Level for block broadcast_19 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
| 2017-10-18 07:30:48 INFO NewHadoopRDD:54 - Input split: HBase table split(table name: flow, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"EhxJA4eZW2uHon2wa4JHdw==","stopRow":"EhxJA4eZW2uHon2wa4JHdw=>","batch":-1,"cacheBlocks":true,"totalColumns":8,"maxResultSize":-1,"families":{"d":["ba","bb","c","h"]},"caching":20000,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: EhxJA4eZW2uHon2wa4JHdw==TrvEBoE/NHtt5hddvafuOg==t+Q75vuTHmtk6jdLMQXYJA==, end row: EhxJA4eZW2uHon2wa4JHdw=>, region location: ip-xx.ec2.internal, encoded region name: 19207f08fce41ba5f363079376ac2e29) | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting local block broadcast_17 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Block broadcast_17 was not found | |
| 2017-10-18 07:30:48 INFO TorrentBroadcast:54 - Started reading broadcast variable 17 | |
| 2017-10-18 07:30:48 DEBUG TorrentBroadcast:58 - Reading piece broadcast_17_piece0 of broadcast_17 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting local block broadcast_17_piece0 as bytes | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting remote block broadcast_17_piece0 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting remote block broadcast_17_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:48 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:48 INFO MemoryStore:54 - Block broadcast_17_piece0 stored as bytes in memory (estimated size 31.8 KB, free 364.8 MB) | |
| 2017-10-18 07:30:48 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_17_piece0 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Told master about block broadcast_17_piece0 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Put block broadcast_17_piece0 locally took 2 ms | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Putting block broadcast_17_piece0 without replication took 2 ms | |
| 2017-10-18 07:30:48 INFO TorrentBroadcast:54 - Reading broadcast variable 17 took 11 ms | |
| 2017-10-18 07:30:48 INFO NewHadoopRDD:54 - Input split: HBase table split(table name: flow, scan: {"loadColumnFamiliesOnDemand":null,"startRow":"EhxJA4eZW2uHon2wa4JHdw==","stopRow":"EhxJA4eZW2uHon2wa4JHdw=>","batch":-1,"cacheBlocks":true,"totalColumns":8,"maxResultSize":-1,"families":{"d":["ba","bb","c","h"]},"caching":20000,"maxVersions":1,"timeRange":[0,9223372036854775807]}, start row: EhxJA4eZW2uHon2wa4JHdw==, end row: EhxJA4eZW2uHon2wa4JHdw==TrvEBoE/NHtt5hddvafuOg==t+Q75vuTHmtk6jdLMQXYJA==, region location: ip-xx.ec2.internal, encoded region name: 24c1c7b25e545cccc44c52cb0428718e) | |
| 2017-10-18 07:30:48 INFO MemoryStore:54 - Block broadcast_17 stored as values in memory (estimated size 471.3 KB, free 364.3 MB) | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Put block broadcast_17 locally took 10 ms | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Putting block broadcast_17 without replication took 12 ms | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Getting local block broadcast_17 | |
| 2017-10-18 07:30:48 DEBUG BlockManager:58 - Level for block broadcast_17 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
| 2017-10-18 07:30:48 INFO RecoverableZooKeeper:120 - Process identifier=hconnection-0x3f2b3db connecting to ZooKeeper ensemble=localhost:2181 | |
| 2017-10-18 07:30:48 INFO ZooKeeper:438 - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@fea5eec | |
| 2017-10-18 07:30:48 INFO RecoverableZooKeeper:120 - Process identifier=hconnection-0x7a0a0536 connecting to ZooKeeper ensemble=localhost:2181 | |
| 2017-10-18 07:30:48 INFO ZooKeeper:438 - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7b86a54c | |
| 2017-10-18 07:30:48 INFO ClientCnxn:975 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) | |
| 2017-10-18 07:30:48 INFO ClientCnxn:852 - Socket connection established to localhost/127.0.0.1:2181, initiating session | |
| 2017-10-18 07:30:48 INFO ClientCnxn:975 - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) | |
| 2017-10-18 07:30:48 INFO ClientCnxn:852 - Socket connection established to localhost/127.0.0.1:2181, initiating session | |
| 2017-10-18 07:30:48 INFO ClientCnxn:1235 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15f2bd51ca50015, negotiated timeout = 40000 | |
| 2017-10-18 07:30:48 INFO ClientCnxn:1235 - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x15f2bd51ca50016, negotiated timeout = 40000 | |
| 2017-10-18 07:30:48 INFO TableInputFormatBase:178 - Input split length: 232 M bytes. | |
| 2017-10-18 07:30:48 INFO TableInputFormatBase:178 - Input split length: 232 M bytes. | |
| 2017-10-18 07:30:54 INFO ConnectionManager$HConnectionImplementation:1720 - Closing zookeeper sessionid=0x15f2bd51ca50012 | |
| 2017-10-18 07:30:54 INFO ClientCnxn:512 - EventThread shut down | |
| 2017-10-18 07:30:54 INFO ZooKeeper:684 - Session: 0x15f2bd51ca50012 closed | |
| 2017-10-18 07:30:54 DEBUG TaskMemoryManager:194 - Task 20 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@22ed7bcd | |
| 2017-10-18 07:30:54 INFO Executor:54 - Finished task 0.0 in stage 18.0 (TID 20). 2407 bytes result sent to driver | |
| 2017-10-18 07:30:54 INFO ConnectionManager$HConnectionImplementation:1720 - Closing zookeeper sessionid=0x15f2bd51ca50016 | |
| 2017-10-18 07:30:54 INFO ClientCnxn:512 - EventThread shut down | |
| 2017-10-18 07:30:54 INFO ZooKeeper:684 - Session: 0x15f2bd51ca50016 closed | |
| 2017-10-18 07:30:54 DEBUG TaskMemoryManager:194 - Task 23 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@53769fb7 | |
| 2017-10-18 07:30:54 INFO Executor:54 - Finished task 1.0 in stage 23.0 (TID 23). 2407 bytes result sent to driver | |
| 2017-10-18 07:30:54 INFO ConnectionManager$HConnectionImplementation:1720 - Closing zookeeper sessionid=0x15f2bd51ca50015 | |
| 2017-10-18 07:30:54 INFO ZooKeeper:684 - Session: 0x15f2bd51ca50015 closed | |
| 2017-10-18 07:30:54 INFO ClientCnxn:512 - EventThread shut down | |
| 2017-10-18 07:30:54 DEBUG TaskMemoryManager:194 - Task 22 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@1f84503a | |
| 2017-10-18 07:30:54 INFO Executor:54 - Finished task 0.0 in stage 23.0 (TID 22). 2407 bytes result sent to driver | |
| 2017-10-18 07:30:54 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 24 | |
| 2017-10-18 07:30:54 INFO Executor:54 - Running task 0.0 in stage 24.0 (TID 24) | |
| 2017-10-18 07:30:54 DEBUG Executor:58 - Task 24's epoch is 11 | |
| 2017-10-18 07:30:54 INFO MapOutputTrackerWorker:54 - Updating epoch to 11 and clearing cache | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Getting local block broadcast_20 | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Block broadcast_20 was not found | |
| 2017-10-18 07:30:54 INFO TorrentBroadcast:54 - Started reading broadcast variable 20 | |
| 2017-10-18 07:30:54 DEBUG TorrentBroadcast:58 - Reading piece broadcast_20_piece0 of broadcast_20 | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Getting local block broadcast_20_piece0 as bytes | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Getting remote block broadcast_20_piece0 | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Getting remote block broadcast_20_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:54 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:54 INFO MemoryStore:54 - Block broadcast_20_piece0 stored as bytes in memory (estimated size 3.7 KB, free 364.3 MB) | |
| 2017-10-18 07:30:54 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_20_piece0 | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Told master about block broadcast_20_piece0 | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Put block broadcast_20_piece0 locally took 3 ms | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Putting block broadcast_20_piece0 without replication took 4 ms | |
| 2017-10-18 07:30:54 INFO TorrentBroadcast:54 - Reading broadcast variable 20 took 10 ms | |
| 2017-10-18 07:30:54 INFO MemoryStore:54 - Block broadcast_20 stored as values in memory (estimated size 7.1 KB, free 364.3 MB) | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Put block broadcast_20 locally took 1 ms | |
| 2017-10-18 07:30:54 DEBUG BlockManager:58 - Putting block broadcast_20 without replication took 1 ms | |
| 2017-10-18 07:30:54 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 17, partitions 0-1 | |
| 2017-10-18 07:30:54 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 17, fetching them | |
| 2017-10-18 07:30:54 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:54 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:54 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 17 took 13 ms | |
| 2017-10-18 07:30:54 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:54 INFO ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks out of 2 blocks | |
| 2017-10-18 07:30:54 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms | |
| 2017-10-18 07:30:54 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 0 ms | |
| 2017-10-18 07:30:54 DEBUG TaskMemoryManager:194 - Task 24 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@69194e60 | |
| 2017-10-18 07:30:54 DEBUG TaskMemoryManager:194 - Task 24 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@149c142c | |
| 2017-10-18 07:30:54 INFO Executor:54 - Finished task 0.0 in stage 24.0 (TID 24). 2805 bytes result sent to driver | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 25 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 25.0 (TID 25) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 25's epoch is 12 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Updating epoch to 12 and clearing cache | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_21 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_21 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 21 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_21_piece0 of broadcast_21 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_21_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_21_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_21_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_21_piece0 stored as bytes in memory (estimated size 4.7 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_21_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_21_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_21_piece0 locally took 3 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_21_piece0 without replication took 4 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 21 took 15 ms | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_21 stored as values in memory (estimated size 10.0 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_21 locally took 1 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_21 without replication took 1 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 16, partitions 0-1 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 16, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 16 took 3 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 8 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 9 ms | |
| 2017-10-18 07:30:55 INFO ConnectionManager$HConnectionImplementation:1720 - Closing zookeeper sessionid=0x15f2bd51ca50013 | |
| 2017-10-18 07:30:55 INFO ClientCnxn:512 - EventThread shut down | |
| 2017-10-18 07:30:55 INFO ZooKeeper:684 - Session: 0x15f2bd51ca50013 closed | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 21 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@560099bb | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 1.0 in stage 18.0 (TID 21). 2407 bytes result sent to driver | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 25 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@72d5893f | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 25 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@79fe70e4 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 0.0 in stage 25.0 (TID 25). 2893 bytes result sent to driver | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 26 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 19.0 (TID 26) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 26's epoch is 13 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Updating epoch to 13 and clearing cache | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_22 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_22 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 22 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_22_piece0 of broadcast_22 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_22_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_22_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_22_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_22_piece0 stored as bytes in memory (estimated size 3.7 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_22_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_22_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_22_piece0 locally took 6 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_22_piece0 without replication took 6 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 22 took 30 ms | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_22 stored as values in memory (estimated size 7.1 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_22 locally took 1 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_22 without replication took 1 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 13, partitions 0-1 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 13, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 27 | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 28 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 1.0 in stage 26.0 (TID 28) | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 26.0 (TID 27) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 28's epoch is 14 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Updating epoch to 14 and clearing cache | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 27's epoch is 14 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_23 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_23 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 23 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_23_piece0 of broadcast_23 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_23_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_23_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_23_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_23_piece0 stored as bytes in memory (estimated size 5.1 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_23_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_23_piece0 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_23_piece0 locally took 3 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_23_piece0 without replication took 4 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 23 took 10 ms | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_23 stored as values in memory (estimated size 10.9 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_23 locally took 1 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_23 without replication took 1 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_23 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Level for block broadcast_23 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 15, partitions 0-1 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 15, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 15 took 2 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 1 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 1 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 15, partitions 1-2 | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 0 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 13 took 22 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks out of 2 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 0 ms | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 27 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@626d3129 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 0.0 in stage 26.0 (TID 27). 1991 bytes result sent to driver | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 28 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@4a797f97 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 1.0 in stage 26.0 (TID 28). 2806 bytes result sent to driver | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 29 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 27.0 (TID 29) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 29's epoch is 15 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Updating epoch to 15 and clearing cache | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_24 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_24 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 24 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_24_piece0 of broadcast_24 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_24_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_24_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_24_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 30 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 1.0 in stage 27.0 (TID 30) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 30's epoch is 15 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_24_piece0 stored as bytes in memory (estimated size 3.4 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_24_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_24_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_24_piece0 locally took 3 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_24_piece0 without replication took 3 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 24 took 16 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 19 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 19 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_19 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_19 of size 8888 dropped from memory (free 381950812) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_19_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_19_piece0 of size 4914 dropped from memory (free 381955726) | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_24 stored as values in memory (estimated size 6.1 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_24 locally took 6 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_24 without replication took 6 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_24 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Level for block broadcast_24 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_19_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_19_piece0 | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 14, partitions 1-2 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 14, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 14 took 2 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 14, partitions 0-1 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks out of 2 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 1 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 1 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks out of 2 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 0 ms | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 26 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@74f79e6a | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 26 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@e522d5 | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 30 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@13edf61e | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 19, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 20 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 20 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_20 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_20 of size 7312 dropped from memory (free 381963038) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_20_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_20_piece0 of size 3762 dropped from memory (free 381966800) | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 0.0 in stage 19.0 (TID 26). 2892 bytes result sent to driver | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_20_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_20_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 20, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 21 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 21 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_21 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_21 of size 10256 dropped from memory (free 381977056) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_21_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_21_piece0 of size 4829 dropped from memory (free 381981885) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_21_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_21_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 21, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 18 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 18 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_18_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_18_piece0 of size 4913 dropped from memory (free 381986798) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_18_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_18_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_18 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_18 of size 8888 dropped from memory (free 381995686) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 18, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 1.0 in stage 27.0 (TID 30). 2251 bytes result sent to driver | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 29 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@6ee0877d | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 0.0 in stage 27.0 (TID 29). 4221 bytes result sent to driver | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 31 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 20.0 (TID 31) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 31's epoch is 16 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Updating epoch to 16 and clearing cache | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_25 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_25 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 25 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_25_piece0 of broadcast_25 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_25_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_25_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_25_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_25_piece0 stored as bytes in memory (estimated size 4.7 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_25_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_25_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_25_piece0 locally took 3 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_25_piece0 without replication took 4 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 25 took 18 ms | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_25 stored as values in memory (estimated size 10.0 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_25 locally took 1 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_25 without replication took 3 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 12, partitions 0-1 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 12, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 12 took 20 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 1 ms | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 31 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@4e70fb17 | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 31 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@241f1a17 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 0.0 in stage 20.0 (TID 31). 2806 bytes result sent to driver | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 32 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 30.0 (TID 32) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 32's epoch is 16 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_26 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_26 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 26 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_26_piece0 of broadcast_26 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_26_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_26_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_26_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_26_piece0 stored as bytes in memory (estimated size 6.8 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_26_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_26_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_26_piece0 locally took 4 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_26_piece0 without replication took 4 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 26 took 14 ms | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_26 stored as values in memory (estimated size 16.4 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_26 locally took 5 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_26 without replication took 5 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 16, partitions 0-1 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 16, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 16 took 3 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 5 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 5 ms | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 33 | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 34 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 21.0 (TID 33) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 33's epoch is 17 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Updating epoch to 17 and clearing cache | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_27 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_27 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 27 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_27_piece0 of broadcast_27 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_27_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_27_piece0 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 1.0 in stage 21.0 (TID 34) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 34's epoch is 17 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_27_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_27_piece0 stored as bytes in memory (estimated size 5.1 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_27_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_27_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_27_piece0 locally took 2 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_27_piece0 without replication took 3 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 27 took 16 ms | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_27 stored as values in memory (estimated size 10.9 KB, free 364.2 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_27 locally took 3 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_27 without replication took 3 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_27 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Level for block broadcast_27 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 11, partitions 1-2 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 11, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 11 took 3 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 1 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 1 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 11, partitions 0-1 | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 1 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 1 ms | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 34 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@7318a7d3 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 1.0 in stage 21.0 (TID 34). 1991 bytes result sent to driver | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 33 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@47a7fcdb | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 0.0 in stage 21.0 (TID 33). 2806 bytes result sent to driver | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 32 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@4bfc75df | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 0.0 in stage 30.0 (TID 32). 2805 bytes result sent to driver | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 35 | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 36 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 22.0 (TID 35) | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 1.0 in stage 22.0 (TID 36) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 36's epoch is 18 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Updating epoch to 18 and clearing cache | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_28 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_28 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 28 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_28_piece0 of broadcast_28 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_28_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_28_piece0 | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 35's epoch is 18 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_28_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_28_piece0 stored as bytes in memory (estimated size 3.4 KB, free 364.2 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_28_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_28_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_28_piece0 locally took 4 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_28_piece0 without replication took 4 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 28 took 16 ms | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_28 stored as values in memory (estimated size 6.1 KB, free 364.2 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_28 locally took 1 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_28 without replication took 1 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 10, partitions 1-2 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 10, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_28 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Level for block broadcast_28 is StorageLevel(disk, memory, deserialized, 1 replicas) | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 10, partitions 0-1 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 10, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 10 took 4 ms | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 10 took 2 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks out of 2 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 1 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 2 non-empty blocks out of 2 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 9 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 9 ms | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 36 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@978358c | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 1.0 in stage 22.0 (TID 36). 3066 bytes result sent to driver | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 24 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 24 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_24 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_24 of size 6224 dropped from memory (free 381936935) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_24_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_24_piece0 of size 3491 dropped from memory (free 381940426) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_24_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_24_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 24, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 25 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 25 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_25_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_25_piece0 of size 4828 dropped from memory (free 381945254) | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 37 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 31.0 (TID 37) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 37's epoch is 19 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Updating epoch to 19 and clearing cache | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_29 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_29 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 29 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_29_piece0 of broadcast_29 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_29_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_29_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_25_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_25_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_25 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_25 of size 10256 dropped from memory (free 381955510) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 25, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_29_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 22 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 22 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_22_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_22_piece0 of size 3768 dropped from memory (free 381959278) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_22_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_22_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_22 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_22 of size 7312 dropped from memory (free 381966590) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 22, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_29_piece0 stored as bytes in memory (estimated size 2.4 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 27 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 27 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_27_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_27_piece0 of size 5233 dropped from memory (free 381969321) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_29_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_29_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_29_piece0 locally took 4 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_29_piece0 without replication took 4 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 29 took 15 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_27_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_27_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_27 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_27 of size 11152 dropped from memory (free 381976137) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 27, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG TaskMemoryManager:194 - Task 35 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@3593d8ad | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing shuffle 14 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing shuffle 14, response is true | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing broadcast 23 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: true to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing broadcast 23 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_23 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_23 of size 11152 dropped from memory (free 381987289) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Removing block broadcast_23_piece0 | |
| 2017-10-18 07:30:55 DEBUG MemoryStore:58 - Block broadcast_23_piece0 of size 5227 dropped from memory (free 381992516) | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_29 stored as values in memory (estimated size 4.2 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_23_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_23_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing broadcast 23, response is 0 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: 0 to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_29 locally took 9 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - removing shuffle 15 | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Done removing shuffle 15, response is true | |
| 2017-10-18 07:30:55 DEBUG BlockManagerSlaveEndpoint:58 - Sent response: true to 172.xxx.xxx.xxx:37499 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_29 without replication took 11 ms | |
| 2017-10-18 07:30:55 INFO Executor:54 - Finished task 0.0 in stage 22.0 (TID 35). 3406 bytes result sent to driver | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 18, partitions 0-1 | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 18, fetching them | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:55 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:55 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 18 took 6 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:55 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms | |
| 2017-10-18 07:30:55 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 1 ms | |
| 2017-10-18 07:30:55 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 38 | |
| 2017-10-18 07:30:55 INFO Executor:54 - Running task 0.0 in stage 34.0 (TID 38) | |
| 2017-10-18 07:30:55 DEBUG Executor:58 - Task 38's epoch is 19 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_30 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Block broadcast_30 was not found | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Started reading broadcast variable 30 | |
| 2017-10-18 07:30:55 DEBUG TorrentBroadcast:58 - Reading piece broadcast_30_piece0 of broadcast_30 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting local block broadcast_30_piece0 as bytes | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_30_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Getting remote block broadcast_30_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:55 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_30_piece0 stored as bytes in memory (estimated size 6.8 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_30_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Told master about block broadcast_30_piece0 | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_30_piece0 locally took 3 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_30_piece0 without replication took 3 ms | |
| 2017-10-18 07:30:55 INFO TorrentBroadcast:54 - Reading broadcast variable 30 took 18 ms | |
| 2017-10-18 07:30:55 INFO MemoryStore:54 - Block broadcast_30 stored as values in memory (estimated size 16.4 KB, free 364.3 MB) | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Put block broadcast_30 locally took 1 ms | |
| 2017-10-18 07:30:55 DEBUG BlockManager:58 - Putting block broadcast_30 without replication took 1 ms | |
| 2017-10-18 07:30:56 DEBUG TaskMemoryManager:194 - Task 37 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@7d0dd705 | |
| 2017-10-18 07:30:56 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 12, partitions 0-1 | |
| 2017-10-18 07:30:56 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 12, fetching them | |
| 2017-10-18 07:30:56 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:56 INFO Executor:54 - Finished task 0.0 in stage 31.0 (TID 37). 5074 bytes result sent to driver | |
| 2017-10-18 07:30:56 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:56 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 12 took 3 ms | |
| 2017-10-18 07:30:56 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:56 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:56 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 0 ms | |
| 2017-10-18 07:30:56 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 0 ms | |
| 2017-10-18 07:30:56 DEBUG TaskMemoryManager:194 - Task 38 release 0.0 B from org.apache.spark.util.collection.ExternalAppendOnlyMap@3e6e38d3 | |
| 2017-10-18 07:30:56 INFO Executor:54 - Finished task 0.0 in stage 34.0 (TID 38). 2805 bytes result sent to driver | |
| 2017-10-18 07:30:56 INFO CoarseGrainedExecutorBackend:54 - Got assigned task 39 | |
| 2017-10-18 07:30:56 INFO Executor:54 - Running task 0.0 in stage 35.0 (TID 39) | |
| 2017-10-18 07:30:56 DEBUG Executor:58 - Task 39's epoch is 20 | |
| 2017-10-18 07:30:56 INFO MapOutputTrackerWorker:54 - Updating epoch to 20 and clearing cache | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Getting local block broadcast_31 | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Block broadcast_31 was not found | |
| 2017-10-18 07:30:56 INFO TorrentBroadcast:54 - Started reading broadcast variable 31 | |
| 2017-10-18 07:30:56 DEBUG TorrentBroadcast:58 - Reading piece broadcast_31_piece0 of broadcast_31 | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Getting local block broadcast_31_piece0 as bytes | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Getting remote block broadcast_31_piece0 | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Getting remote block broadcast_31_piece0 from BlockManagerId(driver, 172.xxx.xxx.xxx, 44833, None) | |
| 2017-10-18 07:30:56 DEBUG TransportClient:140 - Sending fetch chunk request 0 to /172.xxx.xxx.xxx:44833 | |
| 2017-10-18 07:30:56 INFO MemoryStore:54 - Block broadcast_31_piece0 stored as bytes in memory (estimated size 2.4 KB, free 364.3 MB) | |
| 2017-10-18 07:30:56 DEBUG BlockManagerMaster:58 - Updated info of block broadcast_31_piece0 | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Told master about block broadcast_31_piece0 | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Put block broadcast_31_piece0 locally took 4 ms | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Putting block broadcast_31_piece0 without replication took 5 ms | |
| 2017-10-18 07:30:56 INFO TorrentBroadcast:54 - Reading broadcast variable 31 took 10 ms | |
| 2017-10-18 07:30:56 INFO MemoryStore:54 - Block broadcast_31 stored as values in memory (estimated size 4.2 KB, free 364.3 MB) | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Put block broadcast_31 locally took 0 ms | |
| 2017-10-18 07:30:56 DEBUG BlockManager:58 - Putting block broadcast_31 without replication took 0 ms | |
| 2017-10-18 07:30:56 DEBUG MapOutputTrackerWorker:58 - Fetching outputs for shuffle 19, partitions 0-1 | |
| 2017-10-18 07:30:56 INFO MapOutputTrackerWorker:54 - Don't have map outputs for shuffle 19, fetching them | |
| 2017-10-18 07:30:56 INFO MapOutputTrackerWorker:54 - Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://[email protected]:37499) | |
| 2017-10-18 07:30:56 INFO MapOutputTrackerWorker:54 - Got the output locations | |
| 2017-10-18 07:30:56 DEBUG MapOutputTrackerWorker:58 - Fetching map output statuses for shuffle 19 took 3 ms | |
| 2017-10-18 07:30:56 DEBUG ShuffleBlockFetcherIterator:58 - maxBytesInFlight: 50331648, targetRequestSize: 10066329 | |
| 2017-10-18 07:30:56 INFO ShuffleBlockFetcherIterator:54 - Getting 1 non-empty blocks out of 1 blocks | |
| 2017-10-18 07:30:56 INFO ShuffleBlockFetcherIterator:54 - Started 0 remote fetches in 1 ms | |
| 2017-10-18 07:30:56 DEBUG ShuffleBlockFetcherIterator:58 - Got local blocks in 1 ms | |
| 2017-10-18 07:30:56 DEBUG TaskMemoryManager:194 - Task 39 release 0.0 B from org.apache.spark.util.collection.ExternalSorter@514781bc | |
| 2017-10-18 07:30:56 INFO Executor:54 - Finished task 0.0 in stage 35.0 (TID 39). 5074 bytes result sent to driver |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment