Skip to content

Instantly share code, notes, and snippets.

@sebge2emasphere

sebge2emasphere/region-server-6.7secs.txt

Created October 30, 2017 15:21
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save sebge2emasphere/8ba3c5d42a4a68bb2edd848119b187ef to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save sebge2emasphere/8ba3c5d42a4a68bb2edd848119b187ef to your computer and use it in GitHub Desktop.
Download ZIP
Raw
region-server-6.7secs.txt
2017-10-30 15:09:40,443 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 172.31.15.200:56894; # active connections: 3
2017-10-30 15:09:40,470 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.DFSClient: Connecting to datanode 172.31.15.200:50010
2017-10-30 15:09:40,470 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.BlockReaderFactory: BlockReaderFactory(fileName=/user/hbase/data/hbase/meta/1588230740/info/70870e4a202a4ac681d63edc151488e7, block=BP-983401122-172.31.15.200-1508568126354:blk_1073746500_5714): trying to create a remote block reader from a TCP socket
2017-10-30 15:09:40,471 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.BlockReaderFactory: nextTcpPeer: reusing existing peer NioInetPeer(Socket[addr=/172.31.15.200,port=50010,localport=51498])
2017-10-30 15:09:40,471 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] datatransfer.DataTransferProtocol: Sending DataTransferOp OpReadBlockProto: header {
baseHeader {
block {
poolId: "BP-983401122-172.31.15.200-1508568126354"
blockId: 1073746500
generationStamp: 5714
numBytes: 29666
}
token {
identifier: ""
password: ""
kind: ""
service: ""
}
}
clientName: "DFSClient_NONMAPREDUCE_-1070784671_1"
}
offset: 0
len: 29666
sendChecksums: false
cachingStrategy {
}
2017-10-30 15:09:40,472 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.BlockReaderFactory: Closed potentially stale remote peer NioInetPeer(Socket[addr=/172.31.15.200,port=50010,localport=51498])
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2282)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:423)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:818)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:697)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1456)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1702)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1560)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:454)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:922)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:279)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:174)
at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:361)
at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:209)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2132)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5753)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5725)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5730)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2678)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2658)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2640)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2634)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2495)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
2017-10-30 15:09:40,475 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.ProtobufRpcEngine: 80: Call -> ip-172-31-15-200.ec2.internal/172.31.15.200:8020: getServerDefaults {}
2017-10-30 15:09:40,475 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.Client: The ping interval is 60000 ms.
2017-10-30 15:09:40,476 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.Client: Connecting to ip-172-31-15-200.ec2.internal/172.31.15.200:8020
2017-10-30 15:09:40,476 DEBUG [IPC Client (79290250) connection to ip-172-31-15-200.ec2.internal/172.31.15.200:8020 from hbase] ipc.Client: IPC Client (79290250) connection to ip-172-31-15-200.ec2.internal/172.31.15.200:8020 from hbase: starting, having connections 1
2017-10-30 15:09:40,478 DEBUG [IPC Parameter Sending Thread #0] ipc.Client: IPC Client (79290250) connection to ip-172-31-15-200.ec2.internal/172.31.15.200:8020 from hbase sending #173
2017-10-30 15:09:40,481 DEBUG [IPC Client (79290250) connection to ip-172-31-15-200.ec2.internal/172.31.15.200:8020 from hbase] ipc.Client: IPC Client (79290250) connection to ip-172-31-15-200.ec2.internal/172.31.15.200:8020 from hbase got value #173
2017-10-30 15:09:40,481 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.ProtobufRpcEngine: Call: getServerDefaults took 6ms
2017-10-30 15:09:40,481 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.ProtobufRpcEngine: 80: Response <- ip-172-31-15-200.ec2.internal/172.31.15.200:8020: getServerDefaults {serverDefaults { blockSize: 134217728 bytesPerChecksum: 512 writePacketSize: 65536 replication: 1 fileBufferSize: 65536 encryptDataTransfer: false trashInterval: 0 checksumType: CHECKSUM_CRC32C }}
2017-10-30 15:09:40,481 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] sasl.SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /172.31.15.200, datanodeId = DatanodeInfoWithStorage[172.31.15.200:50010,DS-a880eef5-7c7d-47ad-b2df-67a9f0e06072,DISK]
2017-10-30 15:09:40,481 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.BlockReaderFactory: nextTcpPeer: created newConnectedPeer NioInetPeer(Socket[addr=/172.31.15.200,port=50010,localport=51534])
2017-10-30 15:09:40,481 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] datatransfer.DataTransferProtocol: Sending DataTransferOp OpReadBlockProto: header {
baseHeader {
block {
poolId: "BP-983401122-172.31.15.200-1508568126354"
blockId: 1073746500
generationStamp: 5714
numBytes: 29666
}
token {
identifier: ""
password: ""
kind: ""
service: ""
}
}
clientName: "DFSClient_NONMAPREDUCE_-1070784671_1"
}
offset: 0
len: 29666
sendChecksums: false
cachingStrategy {
}
2017-10-30 15:09:40,483 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.DFSClient: Address /172.31.15.200:50010 is local
2017-10-30 15:09:40,484 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Starting read #d6000d7b-b48d-4c06-a63c-9e122fddc19f file /user/hbase/data/hbase/meta/1588230740/info/70870e4a202a4ac681d63edc151488e7 from datanode ip-172-31-15-200.ec2.internal
2017-10-30 15:09:40,484 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] datatransfer.PacketReceiver: readNextPacket: dataPlusChecksumLen = 29666 headerLen = 25
2017-10-30 15:09:40,484 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: DFSClient readNextPacket got header PacketHeader with packetLen=29670 header data: offsetInBlock: 0
seqno: 0
lastPacketInBlock: false
dataLen: 29666
2017-10-30 15:09:40,484 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Reading empty packet at end of read
2017-10-30 15:09:40,485 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] datatransfer.PacketReceiver: readNextPacket: dataPlusChecksumLen = 0 headerLen = 25
2017-10-30 15:09:40,485 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Finishing read #d6000d7b-b48d-4c06-a63c-9e122fddc19f
2017-10-30 15:09:40,485 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Starting read #05a4c55e-6801-49d3-82ff-05e2deac6b5f file /user/hbase/data/hbase/meta/1588230740/info/70870e4a202a4ac681d63edc151488e7 from datanode ip-172-31-15-200.ec2.internal
2017-10-30 15:09:40,485 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Finishing read #05a4c55e-6801-49d3-82ff-05e2deac6b5f
2017-10-30 15:09:40,487 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.DFSClient: Connecting to datanode 172.31.15.200:50010
2017-10-30 15:09:40,487 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.BlockReaderFactory: BlockReaderFactory(fileName=/user/hbase/data/hbase/meta/1588230740/info/c9b54afd4b2b423187f6e98082c6f1b6, block=BP-983401122-172.31.15.200-1508568126354:blk_1073746517_5731): trying to create a remote block reader from a TCP socket
2017-10-30 15:09:40,487 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.BlockReaderFactory: nextTcpPeer: reusing existing peer NioInetPeer(Socket[addr=/172.31.15.200,port=50010,localport=51500])
2017-10-30 15:09:40,488 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] datatransfer.DataTransferProtocol: Sending DataTransferOp OpReadBlockProto: header {
baseHeader {
block {
poolId: "BP-983401122-172.31.15.200-1508568126354"
blockId: 1073746517
generationStamp: 5731
numBytes: 7233
}
token {
identifier: ""
password: ""
kind: ""
service: ""
}
}
clientName: "DFSClient_NONMAPREDUCE_-1070784671_1"
}
offset: 0
len: 7233
sendChecksums: false
cachingStrategy {
}
2017-10-30 15:09:40,488 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.BlockReaderFactory: Closed potentially stale remote peer NioInetPeer(Socket[addr=/172.31.15.200,port=50010,localport=51500])
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2282)
at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:423)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:818)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:697)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:355)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:673)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1456)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1702)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1560)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:454)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:922)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:279)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:174)
at org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:361)
at org.apache.hadoop.hbase.regionserver.StoreScanner.<init>(StoreScanner.java:209)
at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2132)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.initializeScanners(HRegion.java:5753)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5725)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.<init>(HRegion.java:5730)
at org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2678)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2658)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2640)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2634)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2495)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
2017-10-30 15:09:40,488 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] sasl.SaslDataTransferClient: SASL client skipping handshake in unsecured configuration for addr = /172.31.15.200, datanodeId = DatanodeInfoWithStorage[172.31.15.200:50010,DS-a880eef5-7c7d-47ad-b2df-67a9f0e06072,DISK]
2017-10-30 15:09:40,489 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.BlockReaderFactory: nextTcpPeer: created newConnectedPeer NioInetPeer(Socket[addr=/172.31.15.200,port=50010,localport=51538])
2017-10-30 15:09:40,489 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] datatransfer.DataTransferProtocol: Sending DataTransferOp OpReadBlockProto: header {
baseHeader {
block {
poolId: "BP-983401122-172.31.15.200-1508568126354"
blockId: 1073746517
generationStamp: 5731
numBytes: 7233
}
token {
identifier: ""
password: ""
kind: ""
service: ""
}
}
clientName: "DFSClient_NONMAPREDUCE_-1070784671_1"
}
offset: 0
len: 7233
sendChecksums: false
cachingStrategy {
}
2017-10-30 15:09:40,495 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.DFSClient: Address /172.31.15.200:50010 is local
2017-10-30 15:09:40,495 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Starting read #486f0395-fb26-4c25-a800-d5bb98cd39e4 file /user/hbase/data/hbase/meta/1588230740/info/c9b54afd4b2b423187f6e98082c6f1b6 from datanode ip-172-31-15-200.ec2.internal
2017-10-30 15:09:40,495 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] datatransfer.PacketReceiver: readNextPacket: dataPlusChecksumLen = 7233 headerLen = 25
2017-10-30 15:09:40,496 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: DFSClient readNextPacket got header PacketHeader with packetLen=7237 header data: offsetInBlock: 0
seqno: 0
lastPacketInBlock: false
dataLen: 7233
2017-10-30 15:09:40,496 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Reading empty packet at end of read
2017-10-30 15:09:40,496 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] datatransfer.PacketReceiver: readNextPacket: dataPlusChecksumLen = 0 headerLen = 25
2017-10-30 15:09:40,496 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Finishing read #486f0395-fb26-4c25-a800-d5bb98cd39e4
2017-10-30 15:09:40,496 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Starting read #d22df285-43c6-439a-855b-f9dd082d918e file /user/hbase/data/hbase/meta/1588230740/info/c9b54afd4b2b423187f6e98082c6f1b6 from datanode ip-172-31-15-200.ec2.internal
2017-10-30 15:09:40,496 TRACE [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] hdfs.RemoteBlockReader2: Finishing read #d22df285-43c6-439a-855b-f9dd082d918e
2017-10-30 15:09:40,518 DEBUG [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.DFSClient: Connecting to datanode 172.31.15.200:50010
2017-10-30 15:09:40,518 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.BlockReaderFactory: BlockReaderFactory(fileName=/user/hbase/data/hbase/meta/1588230740/info/70870e4a202a4ac681d63edc151488e7, block=BP-983401122-172.31.15.200-1508568126354:blk_1073746500_5714): trying to create a remote block reader from a TCP socket
2017-10-30 15:09:40,518 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.BlockReaderFactory: nextTcpPeer: reusing existing peer NioInetPeer(Socket[addr=/172.31.15.200,port=50010,localport=51534])
2017-10-30 15:09:40,518 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] datatransfer.DataTransferProtocol: Sending DataTransferOp OpReadBlockProto: header {
baseHeader {
block {
poolId: "BP-983401122-172.31.15.200-1508568126354"
blockId: 1073746500
generationStamp: 5714
numBytes: 29666
}
token {
identifier: ""
password: ""
kind: ""
service: ""
}
}
clientName: "DFSClient_NONMAPREDUCE_-1070784671_1"
}
offset: 8265
len: 21401
sendChecksums: false
cachingStrategy {
}
2017-10-30 15:09:40,519 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.DFSClient: Address /172.31.15.200:50010 is local
2017-10-30 15:09:40,519 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.RemoteBlockReader2: Starting read #ef3ddcc3-cf6e-4294-9ed3-92eeea2bff74 file /user/hbase/data/hbase/meta/1588230740/info/70870e4a202a4ac681d63edc151488e7 from datanode ip-172-31-15-200.ec2.internal
2017-10-30 15:09:40,519 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] datatransfer.PacketReceiver: readNextPacket: dataPlusChecksumLen = 21474 headerLen = 25
2017-10-30 15:09:40,519 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.RemoteBlockReader2: DFSClient readNextPacket got header PacketHeader with packetLen=21478 header data: offsetInBlock: 8192
seqno: 0
lastPacketInBlock: false
dataLen: 21474
2017-10-30 15:09:40,519 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.RemoteBlockReader2: Reading empty packet at end of read
2017-10-30 15:09:40,520 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] datatransfer.PacketReceiver: readNextPacket: dataPlusChecksumLen = 0 headerLen = 25
2017-10-30 15:09:40,520 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.RemoteBlockReader2: Finishing read #ef3ddcc3-cf6e-4294-9ed3-92eeea2bff74
2017-10-30 15:09:40,525 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.RemoteBlockReader2: Starting read #67f8f801-9183-4d43-9bf5-694bd19aff21 file /user/hbase/data/hbase/meta/1588230740/info/70870e4a202a4ac681d63edc151488e7 from datanode ip-172-31-15-200.ec2.internal
2017-10-30 15:09:40,525 TRACE [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020] hdfs.RemoteBlockReader2: Finishing read #67f8f801-9183-4d43-9bf5-694bd19aff21
2017-10-30 15:09:40,565 DEBUG [RpcServer.reader=3,bindAddress=ip-172-31-15-200.ec2.internal,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.15.200:56894 because read count=-1. Number of active connections: 3
2017-10-30 15:09:40,622 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 172.31.15.200:56906; # active connections: 3
2017-10-30 15:09:45,176 DEBUG [regionserver/ip-172-31-15-200.ec2.internal/172.31.15.200:16020-SendThread(ip-172-31-15-200.ec2.internal:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x15f6ceb51100030 after 0ms
2017-10-30 15:09:45,641 DEBUG [main-SendThread(ip-172-31-15-200.ec2.internal:2181)] zookeeper.ClientCnxn: Got ping response for sessionid: 0x15f6ceb5110002f after 0ms
2017-10-30 15:09:46,821 DEBUG [RpcServer.reader=4,bindAddress=ip-172-31-15-200.ec2.internal,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 172.31.15.200:56906 because read count=-1. Number of active connections: 3
2017-10-30 15:09:47,151 TRACE [LeaseRenewer:hbase@ip-172-31-15-200.ec2.internal:8020, clients=[DFSClient_NONMAPREDUCE_-1516524792_1], created at java.lang.Throwable: TRACE
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1693)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1658)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:493)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:489)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:504)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1119)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1095)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:374)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:738)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:703)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:610)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:537)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.getWAL(DefaultWALProvider.java:132)
at org.apache.hadoop.hbase.wal.WALFactory.getMetaWAL(WALFactory.java:260)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:1901)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
] ipc.ProtobufRpcEngine: 178: Call -> ip-172-31-15-200.ec2.internal/172.31.15.200:8020: renewLease {clientName: "DFSClient_NONMAPREDUCE_-1516524792_1"}
2017-10-30 15:09:47,152 DEBUG [IPC Parameter Sending Thread #0] ipc.Client: IPC Client (79290250) connection to ip-172-31-15-200.ec2.internal/172.31.15.200:8020 from hbase sending #174
2017-10-30 15:09:47,152 DEBUG [IPC Client (79290250) connection to ip-172-31-15-200.ec2.internal/172.31.15.200:8020 from hbase] ipc.Client: IPC Client (79290250) connection to ip-172-31-15-200.ec2.internal/172.31.15.200:8020 from hbase got value #174
2017-10-30 15:09:47,152 DEBUG [LeaseRenewer:hbase@ip-172-31-15-200.ec2.internal:8020, clients=[DFSClient_NONMAPREDUCE_-1516524792_1], created at java.lang.Throwable: TRACE
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1693)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1658)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:493)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:489)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:504)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1119)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1095)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:374)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:738)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:703)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:610)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:537)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.getWAL(DefaultWALProvider.java:132)
at org.apache.hadoop.hbase.wal.WALFactory.getMetaWAL(WALFactory.java:260)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:1901)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
] ipc.ProtobufRpcEngine: Call: renewLease took 1ms
2017-10-30 15:09:47,154 TRACE [LeaseRenewer:hbase@ip-172-31-15-200.ec2.internal:8020, clients=[DFSClient_NONMAPREDUCE_-1516524792_1], created at java.lang.Throwable: TRACE
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1693)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1658)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:493)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:489)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:504)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1119)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1095)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:374)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:738)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:703)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:610)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:537)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.getWAL(DefaultWALProvider.java:132)
at org.apache.hadoop.hbase.wal.WALFactory.getMetaWAL(WALFactory.java:260)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:1901)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
] ipc.ProtobufRpcEngine: 178: Response <- ip-172-31-15-200.ec2.internal/172.31.15.200:8020: renewLease {}
2017-10-30 15:09:47,154 DEBUG [LeaseRenewer:hbase@ip-172-31-15-200.ec2.internal:8020, clients=[DFSClient_NONMAPREDUCE_-1516524792_1], created at java.lang.Throwable: TRACE
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1693)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1658)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:493)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:489)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:504)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1119)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1095)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:374)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:738)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:703)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:610)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:537)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.getWAL(DefaultWALProvider.java:132)
at org.apache.hadoop.hbase.wal.WALFactory.getMetaWAL(WALFactory.java:260)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:1901)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
] hdfs.LeaseRenewer: Lease renewed for client DFSClient_NONMAPREDUCE_-1516524792_1
2017-10-30 15:09:47,154 DEBUG [LeaseRenewer:hbase@ip-172-31-15-200.ec2.internal:8020, clients=[DFSClient_NONMAPREDUCE_-1516524792_1], created at java.lang.Throwable: TRACE
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1693)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1658)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:493)
at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:489)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:504)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1119)
at org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:1095)
at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:90)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:374)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:738)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:703)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:610)
at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:537)
at org.apache.hadoop.hbase.wal.DefaultWALProvider.getWAL(DefaultWALProvider.java:132)
at org.apache.hadoop.hbase.wal.WALFactory.getMetaWAL(WALFactory.java:260)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:1901)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:363)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
] hdfs.LeaseRenewer: Lease renewer daemon for [DFSClient_NONMAPREDUCE_-1516524792_1] with renew id 1 executed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment